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ABSTRACT 



This thesis reviews the decision-making dilemma caused 
by information uncertainty and ambiguity produced during 
crisis situations. Several cross-impact analysis techniques 
are reviewed and evaluated as possible crisis decision-aids. 
Cross-event analysis is selected and extended for demonstra- 
tion in a hypothetical crisis situation involving South 
Asia. The selected technique is operationalized and employed 
in a controlled environment to assess policy response options 
to the hypothetical crisis. The thesis then assesses the 
technique's conceptual limitations, and evaluates its utility 
as a potential crisis decision-aiding methodology. 
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I. 



DECISION-MAKING IN CRISIS SITUATIONS 



A. INTRODUCTION 

Accounts describing foreign policy-making indicate decision- 
makers frequently deal with profuse quantities of intelligence 
information; which may be inconclusive, ambiguous, or 
conflicting. In an analysis of the Pearl Harbor surprise 
attack, Wohlstetter reveals problems resulting from infor- 
mation quantity and unclear meaning. She characterizes the 
situation as one in which, "...the mass of signals grow 
increasingly dense and freighted with ambiguities." [Ref. 1, 
p. 3]. Sorensen points out that a recurring problem facing 
Presidents is that the facts concerning a situation "may 
be in doubt or dispute." [Ref. 2, p. 19]. Schlesinger high- 
lights the ambiguity surrounding the 1963 Vietnam situation 
by observing that the President asked; "Were you two gentle- 
men in the same country?" [Ref. 3, p. 993], upon receiving 
two widely divergent situation assessments from two officials 
sent to Vietnam on a fact-finding mission. This story also 
demonstrates the important role individual perceptions play 
in decision-making. 

Analysts suggest perceptions influence the decision 
process because decision-makers interpret information using 
their insights, expectations, experience, and goals. Sprout 
and Sprout refer to this as the "stateman's psychological 
environment (that is his image, or estimate, of the situation. 
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setting, or milieu) [which] may or may not correspond to 
the operational environment" [Ref. 4, p. 318], Perceptions 
are a key factor in recognizing crisis situations [Ref. 5] . 

Yet, it is in just such situations that information overloads 
and inconsistencies, which may confound a decision-maker's 
perceptions, seem to occur most dramatically [Ref. 6]. 

This chapter develops the concept that a "dilemma" exists 
in decision-making resulting from information overload and 
ambiguity. Establishing the dilemma's relationship to 
crises, and establishing its effect on the decision process 
requires introducing several relevant concepts. These include, 
identifying a basic decision process model, defining the 
concept of crisis, identifying its stress and information 
overload effects on decision-making, and examining factors 
which may reduce the dilemma. The purpose is to establish 
the need for a decision-aiding methodology which can lessen 
the dilemma's impact on the policy-making process. 

B. THE DECISION-MAKING PROCESS 

According to general decision-making theory, as developed 
by Turban and Meredith [Ref. 7], the first step in the 
decision-making process is problem identification.^ The same 



Subsequent steps include: searching for the alternative 

courses of action, evaluating the alternatives, and solution 
selection. These steps were not examined in detail but 
should be kept in mind during the discussion of the approaches 
to foreign policy analysis. 
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source suggests identifying problems through comparing 
desired and actual levels of goal attainment. This pre- 
supposes that organizational goals are explicit, and in a 
form which facilitates measuring their level of accomplishment. 

Such a requirement may be met in product oriented organi- 
zations (capturing a specific share of the market, or 
obtaining a specific profit level) however, it becomes much 

more difficult in organizations which frame goals in terms 

2 

of advancing or protecting the nation's interests. Thus, 
in starting the decision process, the policy-maker immediately 
faces a problem using available information to determine what 
national interests an emerging situation threatens. The 
available information's accuracy greatly influences the pre- 
cision of national interest identification and measurement. 

Turbin and Meredith [Ref. 7] propose using an evalua- 
tion shceme in completing the decision process. The evalua- 
tion scheme requires some technique which predicts alternative 
response outcomes, a means of relating outcomes to goals, 
and a decision rule applicable to response selection. The 
authors also describe factors which complicate the decision 
process . 



2 

This problem directly relates to the requirement for 
a method of system monitoring which would be basic to a 
decision-aiding system. This aspect of the aid was not 
developed extensively in the current study due to its com- 
plexity and the time required to develop, and validate 
indicators. Significant research is being conducted in this 
area by C.A. McClelland and others in the Threat Recognition 
and Analysis Project [Ref. 8]. 
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Complications develop in decision-making when incorpora- 
ting multiple goals, evaluating an indefinite number of 
alternatives, and determining how sensitive outcomes are 
to changes in problem configuration. (Changes in problem 
configuration could result from incomplete or inaccurate 
information. These slight changes may alter a policy alterna- 
tive's impact on overall situational outcomes, and require 
an analysis of the sensitivity of the policy option's effect 
to changing circumstances.) The above problems obviously 
exist in the policy decision process. This study's major 
premise is that an information processing dilemma, beyond 
the range of these problems, further complicates policy 
decision-making in a crisis situation. The characteristics 
defining a crisis situation produce this dilemma. 

C. THE CONCEPT OF CRISIS 

Traditional international relations studies often ex- 
plore the relationship between "crisis" and "normal" inter- 
actions. These works, usually describing events surrounding 
a crisis, give the concept several intuitive interpretations. 

The most common among these view crises as marking "turning 
points" in international relations. Other interpretations 
describe crises as events which occur naturally within the system. 



3 In the Marxist -Leninist perspective, the occurrance of 
international crisis is an expression of the process of world 
revolution. Traditional power theory explains crisis as the 
operation of the balance of power. Here, crisis results from 
changes in the distribution of power, or from policy makers 
applying "field-tests" to determine the status of the balance 
[Ref . 9 ] . 
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In his essay on "The Acute International Crisis," McClelland 

reviews these interpretations and concludes: 

The crises are thought to be connected 

with the international struggles for 

ideas and also for power. They are seen 

as indicators of the state of the 

international system, but are regarded 

as prime operations in the system as 

well. They seem to be related to going 

to war and to staying at peace. [Ref. 9, p. 188]. 

Obviously, crisis is a pivotal concept in international rela- 
tions. The problem is to establish it as a distinct phenomenon 
with recognizable characteristics. 

Recent theoretical approaches to international rela- 
tions studies attempt to operationalize the concept of 
crisis. These approaches are examined in a later section 
developing the dilemma's impact on the policy decision pro- 
cess. The approach advanced by Hermann [Refs. 10 and 11] 
was selected as most relevant to the current study. He des- 
cribes three dimensions which bound a crisis situation. 

1 . High Threat 

The first dimension defining crisis arises because 
the situation poses "a potential hindrance or obstruction to 
some object or state of affairs that a decision-making unit 
is motivated to achieve." [Ref. 10, p. 29]. A situation 
that threatens important goals produces fear that failure 
to make a "good" decision could leave these goals in jeopardy. 
Having accurate and timely information, therefore, becomes 
extremely important. The other dimensions bounding the 
situation make it less likely such information is available. 
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2 . Short Response Time 



Combining with the high threat factor is the real- 
ization that in a short period of time, if no action is 
taken, the situation may change in some major (unfavorable) 
way. This aspect requires reaching a decision quickly, al- 
lowing little time for collecting information. Furthermore, 
short response time does not permit establishing elaborate 
search routines to disclose alternative response options. 

Thus, information currently available forms the basis for 
the decision. While decision-makers may receive more informa- 
tion, its accuracy is questionable, and individual percep- 
tions become more important in determining what policy actions 
are eventually taken. The third dimension limits the stored 
information's utility. 

3 . Surprise 

This final dimension results from the "absence of 
awareness on the part of policy makers that the situation 
is likely to occur." [Ref. 10, p. 30]. This surprise aspect 
may stem from incomplete information, or result from a 
policy-maker's satisfaction with the situation, producing 
a reluctance to believe contrary reports. Surprise makes 
it unlikely advance preparations exist for coping with the 
situation, producing an "ad-hoc" decision process. Surprise 
also prevents serious data-collection which usually precedes 
significant anticipated events [Ref. 11, p. 416]. These 
characteristics produce a dilemma which inhibits effective 
crisis response. 
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D. THE DILEMMA 

In responding to a developing crisis, policy-makers 
face a dilemma dealing with breakdowns in what information 
usually means. As a crisis situation emerge, policy-makers 
receive unanticipated information inputs. These inputs are 
unexpected because they differ from perceptions concerning 
"normal'' information outputs which the situation should be 
generating. Thus, the dilemma's first aspect involves un- 
derstanding seemingly "unique" events. Policy-makers must 
correctly assess this information to realize that a problem 
exists, and to determine its importance. 

As the crisis continues, more unique inputs flow in, 
increasing the information load the policy-maker must pro- 
cess [Ref. 12]. This aspect taxes the decision units' 

4 

ability to sort and rapidly interpret available information. 
Janis and Mann [Ref. 14] identify several compensating proce- 
dures individuals use in coping with information overload. 



4 .... 

A large body of literature exists indicating that in- 
dividuals are sub-optimal information processors. Fisher, 
et al. [Ref. 13] review the literature on individual's in- 
formation processing abilities. Their report indicates that 
unaided human judgements in complex decision tasks are often 
less accurate than formal algorithms. For instance, evidence 
presented indicates that individuals are conservative in 
interpreting data due to misperception and misaggregation. 

This implies that decisions are normally based on a small 
number of information items, thus excess data and effort spent 
in its collection is "wasted". Other evidence, collected 
from "real-world" studies, indicates that decision-makers 
have difficulty using multiple cue information. 

The current study examines the way in which crisis 
situations further reduce decision-makers' capabilities to 
utilize information. The Fisher report indicates that the 
existance of the basic problem is enough to justify a 
decision-aiding system. 
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Many of these inhibit the accurate evaluation of response 
alternatives . 

Evaluation also requires dependable information. Un- 
fortunately, information during a crisis may be inaccurate 
or conflicting due to source unavailability or distortion. 

Many factors might cause this distortion; however, the 
effect is to reduce useful information. The policy maker 
must discriminate between relevant and irrelevant; reliable 
and unreliable information. A crisis situation's high 
threat and time pressure components imapir the discrimina- 
tion ability which information ambuiguity requires. 

One of this study's central propositions is that: Policy- 

makers in a crisis situation must struggle to interpret 
large quantities of unusual data, having ambiguous or con- 
flicting meaning. Because the situation involves high 
threat, the perception develops that the decision is crucial. 
The rapid event flows require accurate information evaluation, 
and allow a limited decision time. The situation's surprise 
characteristic precludes utilizing routine response measures. 

To interpret large quantities of ambiguous data the policy- 
maker must rapidly and efficiently focus all available informa- 
tion and expertise on the situation. 

Reviewing current approaches to foreign policy analysis 
indicates that this dilemma affects the entire policy-making 
process. This review reveals the requirement for a real 
time decision-aiding methodology which can reduce the infor- 
mation dilemma's impact on policy formulation. 
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E. FOREIGN POLICY ANALYSIS 



Traditional foreign policy studies developed the found- 
ations for more systematic approaches in identifying threats 
to national interests (crises) . Hermann [Ref. 10] and 
McClelland [Ref. 9] review the intuitive interpretations of 
crisis found in descriptive international relations studies. 
The early works these authors review describe crises as 
"turning points" in international relations. 

Crises were believed to result from competing national 
interests, or as situations which developed from a series 
of events a nation viewed as somehow posing a threat to its 
interests. The effort to develop a more exact understanding 
of foreign policy and crises produced two broad analytic 
approaches. These approaches can be classified "systemic" 
and "decisions;" both advance theoretical arguments and 
research findings revealing the policy maker's dilemma. 

Parker [Ref. 15] describes the tenets of these two approaches 
and their major contributions. 

A third approach stems from applying information theory 
to foreign policy analysis, contributing new insights about 
communications flows in policy-making. This technique 
attempts to quantify concepts such as uncertainty and "noise. 

1. Systems Approach 

The systems approach views crisis as an indication 
of the overall state of the system, directly related to 
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system stability. 5 The connection to the traditional 
intuitive interpretations is obvious. From this viewpoint it 
can be argued the occurrence of crises can be used to iden- 
tify the emergence of a problem situation. 

An alternative view of crisis in systems studies 
proposes that it is a prime operation in the system itself. 
This view utilizes March and Simon's bureaucratic process 
model [Ref. 17]. In this model, organizations (modernizing 
societies) strive for stability in their environment. Recog- 
nizing a problem as having been dealt with previously, the 
organization tends to employ repetitive responses. As a 
result the organization establishes routine procedures when- 
ever possible. Crises, in this model, are complexes of 
events overflowing normal processing channels, forcing new 
outputs , which return to the environment producing new 
inputs until new solutions are found [Ref. 9] . This sequence 
demonstrates how information ambiguity and uncertainty impact 
the decision process. 

2 . Decisions Approach 

The second major approach to foreign policy analysis 
suggests that a stimulus-response relationship exists be- 
tween crisis and decision [Ref. 11]. This approach highlights 



5 McClelland [Ref. 9] derives this interpretation from 
Hans J. Morgenthau's Politics Amoung Nations . Price [Ref. 
16] proposes that system stability and decision unit flexi- 
bility combine to define the range of options decision- 
makers consider in a crisis situation. 
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the importance that decision-makers' perceptions play in 
responding to incoming data . ^ 

The decisions approach identifies crisis as a sit- 
uational variable affecting the decision process in two 
ways. These result from crisis' stress inducing character- 
istics, and their impact on information processing. These 
factors are closely interrelated but, for reasons of clarity, 
are explored separately in this study. 

Hermann [Ref. 10] proposes that crises are situations 
involving high threat, allowing short response time, and 
surprising the decision unit. Holsti [Ref. 6] , and other 
authors [Ref. 14], present psychological evidence demonstrating 

7 

that these characteristics are stress inducing. The findings 
indicate that crisis induced stress reduces decision-making 
capabilities, especially those involved in identifying, 
sorting, processing, and interpreting information. 

Foreign policy studies [Ref. 6], and field experi- 
ments [Ref. 10], give credence to these conclusions. These 



£ 

Sprout and Sprout fully develop this concept in their 
works [Ref. 4 and 18]. Holsti, et al, [Ref. 5] modeled the 
process utilizing this factor. The model included: (1) the 

stimulus (meaning policy) applied by the initiating state, 

(2) the perception of the stimulus by the receiving state's 
decision unit, (.3) the receiving state's policy response, and 
(4) response perception by the initiating state's decision unit. 

7 A high degree of threat to important values produces 
anxiety and overreaction. Long working hours required by 
short response times, and the perception of that pressure, 
magnify stress. Unanticipated, novel situations (surprise) 
are generally more threatening to decision-makers [Ref. 6]. 
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findings indicate that policy makers face a dilemma inter- 
preting and using information throughout the entire decision 
process. Finally, indications are that stress is self- 
reinforcing, suggesting that the dilemma increases as the 
situation continues [Ref. 6]. 

These studies reveal how stress affects the decision 
process by reducing the decision unit's ability to process 
information. One study went further and examined informa- 
tion volume produced during the crisis leading to World 

War I [Ref. 6] . Holsti's findings indicate that both infor- 

8 

mation flow and uncertainty rose during the pre-war period. 
This increases both stress and information overlaod [Ref. 14]. 
3 . Information Theory 

Using information theory in foreign policy analysis 
is a relatively new approach which clarifies the relation- 
ship between uncertainty and stress. It measures information 
levels and uncertainty contained in transmitted messages. 
Studies using this technique report findings which relate to 
interpreting incoming intelligence information. 

Lee [Ref. 21] explains information measurement, and 
demonstrates that "noise" in the system produced by erroneous 



g 

Other studies have explored the role of uncertainty 
in the occurrence of war. Singer, et al. [Ref. 19] conclude 
that the changing role of uncertainty accounts for the differ- 
ent time spans for which their models predicted the occurrence 
of conflicts. An attempt to formalize the procedure used 
in the Singer study demonstrated the potential importance 
of uncertainty in the decision process [Ref. 20]. 
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or conflicting information reports significantly reduces 
a message's information value. McClelland [Ref. 22] uses 
information theory in developing a measure of uncertainty, 
which he employed in studying the Berlin question. This 
value represents uncertainty levels present in a given sit- 
uation. This uncertainty reflects analysts' efforts to 
respond to the other side's actions. Uncertainty, measured 

in terms of H-Rel values, increased fairly sharply during 

9 

"obvious" crisis phases in the Berlin problem. 

These findings demonstrate that policy decisions are 
made under uncertainty and increased information volume. 

This situation strains the decision-maker's ability to cope 
with an unfolding situation. Threat, time pressure, and 
surprise combine to increase stress, and further degrade 
decision-making capabilities. 

The preceding discussion explored the information 
dilemma facing policy makers. This dilemma involved three 
aspects: (1) responding to unexpected information, (2) coping 

with increased information volume, and (3) interpreting 
ambiguous, conflicting reports. Because the dilemma reduces 
decision-making capabilities, the requirement exists for 
a decision-aid to reduce its impact on the decision process. 



9 

H-Rel is defined as the ratio of absolute uncertainty 
(some knowledge of performance) to maximum uncertainty (no 
performance data known). As this ratio approaches 1.00, 
it suggests that the behaviors have shown increasing signs 
of disorder liness , or indicates a large amount of "variety" 
in event emissions [Ref. 22, p. 172]. 
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To be useful such an aid must perform several 
functions . 

F. REDUCING THE DILEMMA 

The problems of multiple goals, evaluating an 
indefinite number of options, possible slight changes 
in problem configuration, and the need to incorporate new 
information complicates the policy decision process [Ref. 

7]. A useful decision-aid must, therefore, react to changing 
circumstances and incorporate diverse inputs. 

Information relevant to a particular situation is 
often incomplete, outdated, or inaccurate. The decision- 
maker must, therefore, be able to draw on data other than 
information about the immediate situation [Ref. 11] . The 
perceptions and judgments of specialists in the field might 
be one source of such data. The decision-aid should focus 
specialists' expertise and knowledge to extract meaning from 
incomplete information. 

A true decision-aiding system - as opposed to an infor- 
mation processing system - should help reduce the problem. 

It should facilitate problem identification, aid in enumera- 
ting options, and help determine their potential outcomes. 

It should also provide the means to evaluate outcomes, and 
indicate optimum responses. 

The short response time inherent in a crisis situation 
requires rapid yet considered action. To reduce decision 
time and allow thorough evaluation, the decision-aid should 
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be computer adaptable. This capability would permit 
obtaining interactions from numerous, geographically separate 
analysts and specialists. It would also provide the capa- 
bility for conducting sensitivity analysis, and hard copy 
results for later analysis. 

G. OBJECTIVES 

This research project will develop and evaluate a poten- 
tial real time decision-aiding technique to examine policy 
options for response in a crisis situation. A decision 
analysis methodology which deals with alternative evaluation 
under uncertainty will be selected as the basis for the 
potential decision-aid. 

After determining the methodology's information input 
requirements, working assumptions, limitations, and alterna- 
tive operation procedures, the methodology will be adapted 
into a policy analysis format. Designing a policy analysis 
format requires structuring the methodology to identify and 
rank national objectives affected by a specific problem, to 
identify possible response options supporting those objectives, 
to determine each option's contribution to achieving those 
objectives, and to evaluate their relative merit. 

The modified technique will then be demonstrated in a 
controlled environment- A study group, representing experts, 
will be presented with a hypothetical crisis situation 
in which the objective is to determine an effective response. 
The technique will be used to guide the group in their 
consideration of the problem, and to correlate their inputs. 
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Based on this demonstration, the analytic technique 
will be evaluated on its effectiveness in structuring and 
obtaining a solution to the problem. The technique's util- 
ity will be assessed regarding its flexibility, ease of 
application, performance, and potential for computerization. 

H. CONCLUSIONS 

A crisis is defined by three "situational" boundaries 
[Refs . 10 and 11] . 

1. High threat to decision unit goals. 

2. Short response time available. 

3. Surprising the decision unit by its occurrance. 

These dimensions impact the decision process producing three 
complicating factors which reduce a decision-maker's already 
questionable abilities [Ref. 13] to perform effectively when 
it is vitally important to do so. 

These factors are [Refs. 6 and 22]: 

1. Stress 

2. Uncertainty 

3. Information Overload 

Increases in the levels of these factors, combined with poor 
information processing capabilities, produces a dilemma in 
interpreting information concerning a rapidly evolving situa- 
tion . 

This dilemma reduces the decision-maker's ability to: 

1. Identify relevant useful data. 

2. Correlate and interpret infromation. 

3 . Use this information to draw accurate inferences , 
and make responsible judgments. 

4. Determine appropriate response alternatives, and 
evaluate their utility. 

Make a logical solution selection. 
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Reducing this dilemma to improve decision-making 
abilities requires a decision-aid which is: 

1. Flexible enough to incorporate multiple goals, 
diverse options, and incomplete information. 

2. Draws on the "stored information" of an expert's 
perceptions to interpret the situation. 

3. Quickly focuses the expertise and knowledge of 
numerous specialists on a situation to evaluate 
possible responses. 

4. Systematically evaluates alternatives and indicates 
an optimal response. 

5. Is computer adaptable. 

Subsequent chapters of this study will identify and 
examine an existing decision analysis technique, develop 
it into a possible policy analysis tool, demonstrate its 
application, and evaluate its utility. 
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II. CROSS-IMPACT ANALYSIS 



A. INTRODUCTION 

The previous chapter identified problems facing decision- 
makers operating in an environment characterized by uncertainty 
and ambiguous information. In a social or political environ- 
ment, moving from problem awareness to effective control or 
corrective actions requires that decision-makers understand 
the complex relationships operating in the environment. Such 
understanding is inhibited when a problem's components are 
defined and related through imprecise concepts and incomplete 
theories . 

As indicated earlier, social and political problems are 
not easily adaptable to systematic "scientific" analysis. 

In many cases, the problem elements are not adequately defined 
nor operationalized. Furthermore, no adequate theory exists 
which describes the complex relationships among problem ele- 
ments and indicators. These difficulties are apparent in 
the attempts to construct theories, define variables, and 
identify reliable indicators for crises reviewed in Chapter 
One . 

Socio-economic and political problems are defined using 
imprecise concepts based on a limited understanding of the 
relationships among the variables involved. For this reason, 
conducting systematic decision analysis requires an alterna- 
tive approach to problem specification. The basis for one 
alternative approach is suggested by Helmer: 
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. . .we can either throw up our hands 
in despair and wait until we have an 
adequate theory enabling us to deal 
with socio-economic problems as con- 
fidently as we do with problems in 
physics and chemistry, or we can 
make the most of an admittedly unsat- 
isfactory situation and try to obtain 
the relative intuitive insights of 
experts and then use their judgments 
as systematically as possible. 

[cited in Ref. 23, p. 2] 

The Delphi Technique was an early approach to employing 
subjective judgments in problem evaluation. It was sub- 
jected to several criticisms, one being that it produced 
"linearly independent" estimates which failed to account 
for potential interactions among the events being considered 
[Ref. 24]. Cross-Impact analysis is a Delphi extension 
intended to overcome this problem. 

This chapter will present a non-technical explanation of 
the fundamental notions underlying cross-impact analysis. 

It will identify and classify contentious issues which must 
be resolved before the technique can be used. This discussion 
is intended to provide a basis for systematically comparing 
the various approaches to conducting cross-impact analysis. 
Specific areas which will be explored include: (1) the tech- 

nique's basic concepts, (2) its information requirements 
and their interpretations, (3) the computational procedure 
used, and (4) the technique's outputs and possible applica- 
tions . 

In their review of the literature on decision-making, 
Fischer, et. al. examine several decision-aiding technologies 
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which use subjective estimates. These technologies follow 
a "decomposition" approach involving six tasks: 

1. Recognizing that a decision 
problem exists. 

2. Identifying the possible 
courses of action. 

3. Constructing a probabilistic 
model of the decision-making 
environment . 

4. Constructing a model to eval- 
uate the possible consequences 
of each available action. 

5. Selecting a course of action. 

6 . Implementing the alternative 
selected. 

[Ref. 13, p. 43-44] . 

Because steps 1, 2, and 6 involve understanding the 
creative and organizational processes inherent in decision- 
making, most decision-aiding technologies concentrate on 
steps 3, 4, and 5. As this study will show, cross-impact 
analysis fits in well with these techniques . ^ Cross-impact 
analysis is similar to other subjective techniques in that: 
it develops an event set for defining the problem environment, 
it constructs a probabilistic model of that environment, and 
it uses the model to evaluate possible courses of action. 



It should be noted that, on occasion, an entire commun- 
ity's evaluation can "miss the mark". The recent events in 
Iran provide an example. Reportedly the entire intelligence 
community was surprised at the situation's rapid deteriora- 
tion (two crisis dimensions) [Ref. 25]. A large body of 
psychology and organizational information processing litera- 
ture addresses the causes of this phenomenon. These causes 
are outside this study's scope, which attempts to develop 
and evaluate an aid for reducing the information dilemma. 
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B. THE CONCEPT 



Cross-impact analysis is a technique which attempts to 
identify a given situation's possible outcomes by quantifying 
the linkages between potential developments. This is accom- 
plished by systematically assessing how any particular event 
impacts all other relevant events. The technique and its 
various applications are based on the realization that most 
events are connected in some way [Ref. 26] . As explained 
by Gordon and Hayward: "it is hard to imagine an event 

without a predecessor which made it more or less likely or 
influenced its form - or one which, after occurring, left 
no mark." They term these possible interrelationships "cross- 
impacts" [Ref. 26, p. 101]. 

A second principle behind this technique is, that due 
to the nature of the "inexact sciences", the only possible 
approach to quantifying these interrelationships is to 
employ an expert's subjective judgments as a "substitute 
for the exact laws of causality found in the physical sciences." 
[Ref. 24, p. 134]. When the technique is employed, estimates 
are obtained for three event aspects hypothesized to govern 
possible interactions among future developments: (1) Mode - 

whether an event "enhances" or "inhibits" another; C2) Strength - 
how strong an impact one event has on another; and (3) Time 
Lag - how long before an event's influence is felt by other 

t 

events. These aspects are developed in Gordon, Enzer, and 
Martino [Refs. 26, 27, and 28]. While latter cross-impact 
approaches do not assess these aspects explicitly, understanding 
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the interaction concept is important to understanding the 
intention and approach behind the analysis. 

As a corollary to using subjective estimates, group 
opinions are usually obtained on the basis that: "The 

accuracy of a group judgment is always greater than (or at 
least equal to) the average accuracy of the individual 
judgments.” [Dalkey, cited in Ref. 29, p. 140], 11 Fischer, 
et. al. report several other studies which support this 
conclusion [Ref. 13, p. 50]. 

C. REQUIREMENTS FOR CONDUCTING CROSS-IMPACT ANALYSIS 

To develop a probabalistic model of the problem environ- 
ment using cross-impact analysis requires that four proce- 
dures be accomplished. The first three encompass the sub- 
jective estimation process and include: (1) specifying an 

event set which defines the environment, (.2) estimating the 
initial occurrence probabilities of those events, and (3) 
estimating their interactions with each other (or their 
cross-impacts). The final procedure manipulates the model 
formed by these estimates to produce the data with which 
possible courses of action are evaluated. This step requires 
(4) developing an algorithm to use in the manipulation process. 
These procedures will be explained in detail below. 



There are a number of subjective measurement techniques 
which depend upon aggregating group estimates. Subjective 
measurement, aggreation procedures and applications are dis- 
dussed in Torgerson, Miller, and Coombs [Refs. 30, 31, and 3 2 J . 
The commonly used, group-based subjective measurement techniques 
rely heavily on central tendencies (i.e., means and standard 
deviations) in order to determine judgmental scores. 
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1 . Event Set Specification 



The first process undertaken in cross-impact analy- 
sis is developing an event set. This is "a set of possible 
events which are thought to be important to the issue being 
explored" [Ref. 33, p. 33]. The event set may be obtained 
in any manner: delphi, conferences, literature searches, 

consultations with experts, analyst specification, and 
so forth. A single event may address any subject, and the 
entire set may address any combination of subjects such as, 
"potential technological developments, a list of social or 
political developments, a combination of these, or something 
entirely different." [Ref. 34, p. 344]. 

As broad and loose as these requirements may seem, 
the means by which the event set was formulated in several 
of the demonstrations discussed in this study has been criti- 
cized as faulty [Ref. 29, p. 137]. Therefore, it seems 
relevant to review recommendations for accomplishing this 
process . 

Enzer suggests candidate events be developed from the 
likely consequences of inaction, or allowing the "status-quo" 
to continue [Ref. 27, p. 50]. He also proposes several 
criteria for including candidate events in the analysis: 

(1) the development must be important 
to the subject being evaluated, in 
the sense that the event should 
have some impact on an action or 

a decision that is being contem- 
plated. 

(2) only those important developments 
whose outcome is uncertain (that 



is, those that may occur as 
opposed to those that almost 
certainly will or will not occur) 
should be included in the set 
being analyzed. 

(3) if the event set is to be highly 
limited it is also desirable for 
the items to be causally re- 
lated to each other in some 
way. [Ref. 35, p. 345] 12 

(4) Developments or events as referred 
to here denote specific items 
which may occur in the future. 

They are 'specific' in the 

sense that their occurrence 
or non-occurrence can be 
rigorously determined. 

[Ref. 27, p. 50] . 

Mitchell and Tydeman propose several rules for event 
formulation. Their purpose is to reduce possible respondent 
misinterpretation and thereby reduce response variation. 
These rules will be followed in the application section of 
this study. 

(a) Except where the cause is part 
of the event to be studied, 
events should not be stated in 
the form 'A occurs because of B ' . 

(b) Events should be single rather 
than multiple, i.e., 'A occurs' 
rather than r A,B, and C occur'. 

(c) Events should, where possible, 
be definite and specific rather 
than vague and general, e.g.,' 



12 Enzer develops the requirement of being "causally related" 
through the concept of "coupling" . He describes three types 
of event coupling, "totally uncoupled", "coupled", and totally 
included". Of these, totally uncoupled events are felt to 
be of little importance in the analysis, coupled events 
shouLd be analyzed, and included events may be analyzed if 
their impacts are not estimated more than once [Ref. 27, p. 52] . 
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'Costs will increase more than 
20% in real terms' rather than 
'Costs will increase signific- 
antly’ . [Ref. 29, p. 136] . 

Formulation rules are necessary to insure that "the group 
involved reach agreement and understanding on the specifica- 
tion and wording of the event set." [Ref. 36, p. 333]. 

Although it is desirable that the event set completely 
define the problem under consideration, this is rarely possi- 
ble. The uncertainties in the problem environment, and the 
complex issues involved could necessitate specifying many 
events. Since the possible first order interactions between 
events increases as the square of the number of events speci- 
fied, estimating their interactions would quickly become 
extremely time consuming [Ref. 34]. Therefore, completeness 

13 

is often obtained by fiat. Since it is almost impossible 
to specify an event set which includes every conceivable 
development most analysts limit the event set to a manage- 
able size and attempt to include the most important or signi- 
ficant events. In such cases, events to be included in the 
analysis are selected from a large list of "candidate" events 
on the basis of subjective importance, desirability, or 
impact ratings. These procedures will be described in the 
next chapter. 



12 This limitation's implications have been examined in 
several cross-impact approaches. The developers (Enzer , Turoff, 
and Mitchell and Tydeman) suggest different methods to iden- 
tify the most important events and to retain the most sensi- 
tive or influential ones for analysis. In this manner, it is , 
felt that the most serious aspects of a problem are considered. 
Furthermore, it is always possible to expand the event set 
as circumstances dictate. Thus, obtaining a complete event 
set by "fiat" is felt to be the only practicable method, 
and does not impose severe limitations. 
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Initial Probabilities 



Once the event set is established and understood by 
the respondents, the next procedure obtains estimates for 
each event's "initial” likelihood of occurrence. These esti- 
mates are variously termed individual, initial or marginal 
. . . 14 

probabxlitres . All refer to a particular event's likeli- 
hood of occurring by some specified date. 

As with event specification, Enzer suggests that 
these estimates should be developed from an environment which 
might evolve naturally from the current situation [Ref. 35]. 
This interpretation recognizes that developments are typically 
seen in their total environment, and not in isolation. 

This viewpoint poses a conceptual problem with cross- 
impact analysis which must be resolved before the technique 
can be operationalized. If events are viewed in their total 
environment when their initial probabilities are estimated, 
do these estimates include a consideration of the occurrence 
of other likely events in the set? In other words, in esti- 
mating the initial probability that event A will occur, do 
respondents implicitly consider the impact on A of another 
event (say B) if the other event also occurs? Each approach 
to cross-impact analysis must deal with this problem, and 



^Different terms are applied by the various cross-impact 
analysis approaches to describe the technique's basic 
concepts. In this study these will be identified as they 
occur, and thereafter the most descriptive, or logically 
consistent term will be used. 
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it will be considered more fully while examining specific 
approaches . 

A reasonable response to this problem appears to 
be that estimators do take the interrelationships among 
events into consideration, but only incompletely and not 
systematically. Therefore, some modification is necessary 
to obtain "consistent" estimates [Ref. 34, p. 342] . 

3 . Cross- Impacts 

The next process obtains "A qualitative description 
of the effect of the occurrence (or non-occurrence) of one 
event on the likelihood of occurrence of the other events 
in the set." [Ref. 27, p. 51]. These are subjective esti- 
mates concerning how an event's occurrence would change the 
probability of every other event's occurrence. These are 
binary estimates, dealing with the interactions between two 
events only. 

In eliciting these estimates, respondents are told 
to assume certain occurrence (or non-occurrence) for an 
event and then assess its impact on each remaining events. 
Rules for event impact assessment state that the events are 
assumed to happen only once (i.e., non-repetitive) , that 
they do not have to happen at all, and that they do not 
occur simultaneously [Ref. 36]. 

These estimates have been termed "conditional" or 
"causal" probabilities. This conflict in terms stems from 
another conceptual problem in cross-impact analysis, relating 
directly to the problem of obtaining consistent estimates. 
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The basic problem is determining how the cross-impacts 
should be interpreted. One interpretation is that the esti- 
mates are conditional probabilities in the "classical" 
sense (i.e., the probability of event A given event B has 
occurred, P(a|b)), or are used to derive conditional proba- 
bilities [Ref. 37]. Accepting this interpretation permits 
using Bayesian probability relationships to test estimate 
consistency. Several developers have derived conditions which 
are intended to insure consistency and can be used to com- 
pute probability bounds on subsequent estimates.^ Another 
interpretation is that the estimates are causal, time-dependent 
probabilities to which classical probability rules do not 
apply (i.e., the probability A will occur assuming B occurs 
first, P (A|B f ) ) • In this case Bayesian probability rela- 
tionships have only limited application [Ref. 39]. 

A third interpretation is that the technique attempts 
to obtain subjective estimates for the correlation coeffi- 
cients between events . In this case inconsistency is not 
readily apparent, as when probability values exceed unity 
or become negative. Turoff applies this interpretation in 
his cross-impact approach [Ref. 36] . 

This problem will be examined in detail in the next 
chapter. Initially, however, it appears that using classical 
probability relationships not only permits testing estimates 



^See, for example, Dalkey and Sarin [Refs. 34 and 38] 
for consistency conditions and boundaries. 
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for consistency, but provides relationships for adjusting 
inconsistent estimates, and furnishes a model for their 
manipulation. Without such a model, and recognizing the 
lack of causal knowledge in the social sciences, the tech- 
nique faces problems in interpreting the subjective esti- 

1 6 

mates, and justifying the manipulation algorithms. 

4 . Computational Procedure 

The final cross-impact analysis process employs a 
technique to analyze estimates and provide useful informa- 
tion to decision makers. As noted by Dalkey, given the 
large numbers of possible interactions, 

even if a matrix describing the inter- 
actions is available . . . the task of 
thinking through the implications 
rapidly gets out of hand. Some com- 
putational aid is required to take 
account of the large number of inter- 
dependencies. [Ref. 34, p. 341], 

Frequently, the approach conducts "a Monte Carlo 

sampling of chains of events in which the probability of 

an event in the chain is modified by the cross-impact of 

the previously occurring event in the chain." [Ref. 34, 

p. 341] . This procedure requires an algorithm for calculating 

the cross-impact effects among events. Several methods of 

manipulation and analysis have been proposed and are examined 

in the next chapter. 



16 It also seems that probability values imply that A and 
B are discrete events; whereas correlation coefficients, "r", 
imply that A and B are indicators of underlying variables to_ 
which the events are linked conceptually within a theoretical 
framework. 
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In tiie Monte Carlo process, an event is randomly 
selected, and its occurrence or non-occurrence randomly 
decided based on its initial probability. The remaining 
events' initial probabilities are adjusted using the selected 
algorithm, according to the first event's cross-impacts. 

At this point a second event is selected and decided, based 
on its new probability. The remaining probabilities are 
revised and the process continues until all events have been 
decided . 

Completing the Monte Carlo procedure is termed a 
run. Conducting numerous runs permits calculating revised 
initial probabilities for each event based on the event's 
experimental frequency of occurrence. As an event is decided 
its occurrence of non-occurrence is recorded (either 1 or 0 
respectively) . Revised initial probabilities are then 
determined by dividing the number of times each event 
occurred by the total number of runs conducted. 

Early research using the technique suggested con- 
ducting more than one thousand runs in order to reach sta- 
bility and reduce random variation [Ref. 26] . However, during 
demonstrations as few as twenty runs produced adequate re- 
sults [Ref. 27]. Johnson examined the requirements con- 
cerning how many runs were needed to produce significant 
probability shifts and stabilize inherent random variation 
[Ref. 40] . Using standard statistical techniques he concluded 
that: (1) The mean probability values for events reached 

stability within a small number of runs (< 50) . This implies 
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that the model reached stability, or a consistent point, 
relatively quickly. (2) The standard deviations of modi- 
fied probabilities decreased with increased numbers of runs. 
However, beyond fifty runs, the magnitude of these reduc- 
tions was small. This indicates that stability was reached, 
and that random variation was only slightly reduced by 
increasing the numbers of runs. (3) As more events were 
included in the analysis set, no notable changes appeared 
in the average standard deviations obtained. This indicates 
that the numbers of runs does not have to increase as the 
number of events increases. (4) Furthermore, Johnson observed 
that while it is possible to obtain significant probability 
shifts from a statistical standpoint, there is no assurance 
that such shifts are significant from a policy standpoint 
[Ref. 40, p. 127-130] . 

D. CROSS-IMPACT ANALYSIS SYMBOLOGY 

The different cross-impact approaches, besides using 
various terms to describe basic concepts, have also employed 
different symbologies in their demonstrations. This section 
is presented in an attempt to establish a consistent symbology 
for cross-impact computations. This will simplify comparing 
the different cross-impact approaches in the next chapter. 

1 . Event Set Specification 

Let A, B, C, ... N be events which could develop from 
present circumstances, which seem to be related to each 
other (i.e., are coupled as previously described), and are 
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important to some problem being considered. These events 

were selected and formulated in accordance with the rules 

specified previously. Designating A as event one (E ) , B 

as event two (E~), etc., produces a set of events {E. , E„, 

^ 12 

E ^ i • • • t } . 

2. Estimate Initial Probabilities 

Each event (E-^ through E ) has an associated initial 
probability of occurring by some specified time in the 
future based on the present situation. It is possible for 
experts to estimate this probability (i.e., if present 
developments continue without change, the probability A, (E^) , 
will occur by time T is x) . Designate this probability (of 
event A) as P(E^) and similarly for all events in the set 
(PfEj.), P(E 2 ), P(E 3 ), ..., P(E n )}. Or as P j _ for i = 1,2, 

3 , . . . ,n. 

3 . Estimate Cross- Impacts 

The next step is to estimate the effect on the other 
events of the occurrence of any one event (i.e., if event 
B is certain to occur what is the new probability event A 
will occur?). P(E^|E 2 ) or P(i|j) for i,j = 1,2,3, ...,n 
and j / i. These estimates are obtained for each possible 
event pair (i,j) and designated in the form P (impacted 
event | certain event). 

4 . Computational Procedure 

The result from this series of estimates may then be 
arranged in an n by n matrix. Since the impact of an event 
on itself obviously has no meaning, the diagonal cell by 
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convention contains the events' initial probability. The 

2 

matrix thus contains all n required estimates. 

FIGURE 1 

BASIC CROSS-IMPACT MATRIX 
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Note that P(E.jE 2 ) is not necessarily equal to P(E 2 |E^). 

The matrix is then manipulated using the Monte Carlo 
procedure previously described, or some other procedure may 
be substituted. Using numerous runs permits computing 
revised initial probabilities for each event based on its 
frequency of occurrence during the runs. 



P'(E n ) 



# times E occurred 
n 

# runs conducted 



E. OUTPUTS AND APPLICATIONS 

This section will review the types of analyses conducted 
using the technique and their possible applications. The 
technique's proponents have demonstrated several types of 



analyses using the data generated by the technique. Many 
applications claimed for these outputs seem to address 
effectively the difficulties identified with problem solving 
in the social sciences. These claims seem plausable in the 
context of the authors' arguments, however, final judgment 
on the technique's utility must be reserved until the 
claims can be verified with data from experiments or real 
world applications. These outputs and their supporting 
rationale will be described below, along with proposed 
applications. Specific procedures for conducting these 
analyses will be presented in the next chapter which 
develops the major cross-impact analysis approaches. 

One often cited potential for the technique is the 
learning experience it provides. Due to its systematic 
approach, estimators are forced to develop and clarify 
their opinions regarding the causal linkages coupling 
potential developments in a field or several fields into a 
consistent picture [Ref. 33]. Estimators must be explicit 
when structuring their responses, and are forced to con- 
sider related impacts and overall consistency. This aids in 
organizing and evaluating an individual's views on a com- 
plex problem [Ref. 36] . 

Manipulating the matrix through the Monte Carlo process, 
or other procedures (other procedures will be discussed 
during the review of specific cross-impact approaches), pro- 
duces event sequence scenarios which indicate the various 
ways in which a given situation might evolve. As used in 



cross-impact, analysis the term scenario has a very specific 
connotation. A scenario refers to the event outcome deci- 
sion (either occurrence or non-occurrenc ) made when the 
estimates are manipulated, and the sequence in which the 
events are decided. For a six event set a possible scenario 
would be depicted in the following manner: 

(lr 0, 0, 1, 1, 1,) 

where events 1,4,5, and 6 occurred, and events 2 and 3 did 
not. In this study, subsequent discussions of the cross- 
impact methodology, of specific cross-impact approaches, and 
of the study's demonstration results will use the term 
scenario (or, event sequence scenario) to refer to these 
event occurrence chains. 

Scenario generation provides decision-makers a background 

17 

against which policy options may be evaluated. By speci- 
fying scenarios, and evaluating them using subjective event 
importance ratings, it might be possible to identify obscure 



Sarin discusses scenario generation and its importance 
[Ref. 38]. For a set of n events there are 2 n possible 
scenarios, each with an associated scenario probability, 

P (x ) . Sarin proposes a method for computing these scenario 
probabilities from expert estimates. The scenario probabili- 
ties may also be computed, as were the revised initial event 
probabilities, through the Monte Carlo process. Each Monte 
Carlo run produces one scenario, P (x n ) for that scenario is 
computed by the frequency that it occurs divided by the 
number of scenarios generated. 



P(x ) = 

n 



# x n produced 
# runs 



However, this requires that many Monte Carlo runs be conducted. 
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but important situations from among the vast number of 
possibilities [Ref. 33]. Used alone, scenario generation 
at least indicates to decision-makers the likely consequences 
of inaction. The scenario output is the technique's major 
tool in its most interesting application, policy analysis. 

Through comparing the scenarios (produced by recording 
event sequence outcomes during a Monte Carlo simulation) to 
the cross-impact matrix it is possible to identify "critical" 
events which make a considerable difference in outcomes [Ref. 
28]. This causal analysis also provides "a means of gaining 
insight into key branch points and items on which current 
actions and policies should be focused in order to increase 
the likelihood of achieving a desired outcome." [Ref. 33, 
p. 45]. 

By examining the cross-impact matrix with the scenarios 
it is also possible to identify possible "side effects" pro- 
duced by the decision under consideration. This examination 
could aid the decision-maker in two ways. First, a nagging 
problem which faces decision-makers is to identify additional 
problems which may arise further down the line caused by 
their attempts to respond to the immediate problem situation. 
Determining a decision's possible side effects may reveal 
unanticipated developments which might stem from a particular 
situation or a particular policy option. 

A second advantage offered by exploring a decision's 
side effects is that the examination may indicate less 
expensive means of attaining goals through investment in 
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high-payoff areas which may have seemed unrelated [Ref. 26, 
p. 101], This advantage is reflected in attempts to use 
cross-impact analysis to evaluate resource allocation strat- 
egies. Gordon suggested resource allocation strategies could 
be tested by adjusting the initial probability estimates for 
one or several events to reflect policy decisions in order to 
improve or inhibit their likelihood. These adjustments are 
then manipulated as was the original matrix, and the output 
examined to determine the policy decision's effect on the 
various scenario probabilities observed [Ref. 24] . This 
process is termed sensitivity analysis, and can be used to 
explore the effects caused by changes in problem configuration. 

Further cross-impact analysis applications derive from 
using the technique to model the relationships operative in 
a given situation [Ref. 4]. Enzer demonstrates that the 
matrix may be used to determine an event's dominance and 
sensitivity. In his approach, the sum of the absolute values 
in a matrix row can be interpreted as a measure of the magni- 
tude of the impact one event's occurrence has on the occur- 
rence likelihood of all other events (its dominance) [Ref. 27, 
p. 54]. As a minimum, this identifies the events which have 
the greatest and least effect, relatively, on the situation. 
Those events with the greatest overall effects are prime 
candidates for intervention through resource allocation. 

Event sensitivity is computed from the algebraic sum 
of the matrix column values. This is a measure indicating 
how responsive an event is to the other events in the set 
[Ref. 27, p. 54]. It reveals events which may be susceptible 
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to change through secondary intervention. It could also 
indicate those events which most readily indicate changes in 
the system, and hence, should comprise the components of a 
monitoring mechanism. 

By focusing on the opinion spread registered by the 
experts, Enzer develops an uncertainty measure reflecting the 
degree of estimator uncertainty concerning the cross-impacts. 
This is derived for each event by determining the distance 
between the upper quartile and lower quartile estimate values 
[Ref. 27, p. 54]. This "uncertainty" measure also highlights 
the level of agreement among the experts, and can be inter- 
preted as a "confidence" measure. Such information would be 
an aid in resource allocation, allowing decision-makers to 
avoid investing effort in influencing an event whose impact 
was highly uncertain. If subjective estimates regarding an 
event's importance and desirability are obtained with the 
probability estimates more elaborate analyses are possible. 

These could aid decision-makers in identifying desirable actions 
and areas warranting further investigation [Ref. 27, p. 57]. 

The cross-impact technique also appears amenable to mani- 
pulation with digraph theory, and other techniques used in 
studies of individual and organizational information process- 
ing (esp. information theory) . Digraph theory uses binary 

matrix entries to trace third- and fourth-level interrelation- 

1 3 

ships among events. Turoff incorporated information theory 

1 8 

The principles and methods of digraph theory are 
presented in Harary, et al . [Ref. 42]. Axelrod [Ref. 43], and 
Bonham, et al. [Refs. 44 and 45] have attempted to apply digraph 
theory to international relations theory and problems. 
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in his cross-impact approach to relate estimates for 
occurrence and non-occurrence [Ref. 36]. 

F. RATIONALE FOR SELECTING CROSS -IMPACT ANALYSIS 

The preceding discussion has revealed several aspects of 
cross-impact analysis which make it attractive as a decision- 
aiding methodology for crisis situations. The technique's 
advantages will be summarized below as justification for 
selecting cross-impact analysis for demonstration and evalu- 
ation in this project. Specific aspects of the technique 
discussed include: (1) its subjective method of evaluation, 

(2) its systematic approach, (3) its flexibility, (4) its 
demonstrated application to policy analysis, and (5) its 
potential utility as a learning tool. 

1 . Subjective 

Using expert's subjective evaluations precludes the 

need for a formal theory relating crisis elements, national 

interests, and policy options. The procedure takes into 

account the perceptions and accumulated knowledge of experts 

and decision-makers (which is not easily transferred) and 

uses them to process and interpret imprecise information. 

It minimizes the need to collect precise data on numerous, 

inadequately understood indicators, especially in situations 

19 

where time and stored information is limited. 

19 

This is not intended to suggest that theory construction 
and indicator development are not important. As noted in 
Chapter One, these approaches are required for system moni- 
toring, furthermore, they are necessary to achieve precision 
in the social sciences as indicated by Rossenau [Ref. 46]. 

What is intended is to establish the current need for sub- 
jective estimates in quantitative analyses of policy problems. 
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2 . Systematic Approach 



Because the technique is systematic in its approach, 
it encourages the consideration of the numerous interactions 
which could be overlooked in situations involving short response 
time and stress. In addition, it proceeds logically from 
problem identification through the selection of policy options. 
The subjective estimates obtained during the evaluation process 
are retained so the entire analysis can be replicated at will, 
rather than being lost after the crisis has passed. 

Finally, because it is a systematic procedure cross- 
impact analysis may be conducted using a computer. Most 
applications demonstrated, especially those presented by 
Mitchell and Tvdeman , Turoff, and Sarin [Refs. 29, 36, and 
38] , have utilized or advocated an interactive computer 
system for querying experts. This capability increases 
analysis speed and flexibility, both of which are desirable 
for a decision-aid in crisis situations. 

3. Flexibility 

Cross-impact analysis is not constrained by data 
requirements to evaluating only a few empirically quantifiable 
variables. The variables used in the technique are obtained 
from experts' perceptions concerning a specific problem. If, 
while obtaining subjective probability estimates, it becomes 
apparent some important aspect of the problem has been 
omitted it is possible to include it in the event set. This 
capability is particularly useful in analyzing a crisis 
situation involving uncertainty and incomplete information. 



48 



It allows the event set to be modified in response to 
changing circumstances. Such "last minute" modifications 
are possible because the values used in the analysis are 
subjective, and can be easily revised or expanded. 

Using subjective judgments also permits the approach 
to address any problem for which an appropriate event set 
can be specified. This flexibility is evidenced by the broad 
spectrum of problems to which the technique has been applied. 

4 . Demonstrated Applications 

Several cross-impact studies have been conducted in 
situations relevant to policy analysis. Heuer describes one 
such study in reviewing CIA efforts to use quantitative 
techniques to analyze policy relevant situations [Ref. 41]. 

The approach is future oriented and addresses the potential 
evolution of a current situation. Thus, cross-impact analysis 
attempts to be predictive, conducts sensitivity analysis, and 
indicates degrees of uncertainty. Other developers have 
applied the methodology to problems of evaluating transporta- 
tion alternatives, population trends, and energy options 
[Refs. 26, 33, and 47], Changing values and objectives are 
implicitly considered by using subjective estimates or may 
be explicitly stated in the cross-impact matrix. Finally, 
once the estimates' form and meaning are understood the 
procedure may be used by anyone. For these reasons, cross- 
impact analysis enhances the process of analyzing a situation 
and reaching a policy decision. 
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5 . Learning Tool 



Since respondent estimates can be retained, the 
technique can help close the feedback loop involved in 
assessing the direction in which a situation is moving. This 
can be accomplished by comparing estimates obtained over time 
(T^ ~ T 2^ * a P°licy has been implemented between and T^ r 

then the comparison tests the policy's success. 

By monitoring the emergence of cross-impact matrices 
over time, it might be possible to gain insight into how 
organizations structure their knowledge about the environment. 
Thus, a series of matrices, taken from a particular situation, 
might reveal some mechanics of organizational information 
processing . 

G. EVALUATION CRITERIA 

The preceding overview highlighted cross-impact analysis' 
potential applicability as a policy tool. However, the 
various conceptual problems identified here and by the tech- 
nique's developers have produced several alternative cross- 
impact approaches, each advancing different methods to deal 
with the technique's conceptual problems, and leading to 
confusion regarding which variation of the technique is the 
best to operationalize. 

v 

Due to the confusion, selecting a variation for extension 
as a crisis decision-aid should be approached systematically 
using consistent evaluation criteria. These criteria should 
address each variation's primary components, including its 
assumptions, underlying rationale, and implications. Specific 



50 



implementation problems should be evaluated by examining 
information requirements, computational procedures, and 
methodological approach. The goal is to select a practicable 
approach and develop it into a crisis decision-aid. 

In developing consistent evaluation criteria, several 
considerations must be kept in mind. These stem from the 
characteristics defining crisis, and this project's time and 
computational resource limitations. Based on these considera- 
tions, discussed in the first chapter, the following criteria 
were formulated for comparing the various cross-impact approaches. 

1. Is the event formulation scope adequate ? This 
refers to the subject area addressed by the event set. In the 
intended analysis, events must cover such concepts as "national 
interests" threatened, and specific "policy options" available. 

2 . Can events be rated on importance/desirability ? 

The need for such subjective estimates in sensitivity analysis 
and event set selection has been indicated. This refers to 
whether or not the specific technique includes procedures for 
obtaining and using such estimates. 

3 . What types of analysis can be performed using the 
technique ? 

4 . Is the technique computer adaptable ? 

5 . Are the estimated relationships clearly formulated ? 
This refers to whether or not the procedure used to obtain 
estimates, and the questions asked, are understood by the 
respondents. Are the respondents asked for estimates they 

can reasonably be expected to supply? 
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6 • Axe the estimates consistent, or can consistency 
be obtained ? This refers to the need for estimates which can 
be manipulated using a systematic, rigorous, methodology. 

7 ♦ What algorithm is used to manipulate the estimates ? 
This refers to the procedure by which the cross-impacts are 
"taken into account" [Ref. 34, p. 342] , and how it is structured. 

8- Is the procedure logical/justifiable ? This 
refers to the assumptions made in adopting the algorithm, are 
they reasonable and what limitations do they impose? 

9. How is consistency obtained ? This refers to the 
computational procedure for checking consistency and revising 
estimates into a consistent set. 

10 . How are scenarios generated ? This refers to the 
procedure used to develop event sequence scenarios, how complex 
it is and whether or not it provides a ranking system. 

11 . How many estimates does the technique require ? 

12. What types of computation are performed ? This 
refers to the complexity of the calculations. Are they 
algebraic, quadratic, differentials, or linear programs? 

13 . How many terms are involved in each formula ? 

14 . How many separate calculations are required ? 

This refers to what computational steps are required for 
estimate manipulation, checking consistency, scenario 
generation and analysis. 

These last four criteria are proposed in order to 
identify an approach which can be demonstrated using limited 
computational facilities. This study will replicate the 
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selected approach manually, therefore, the number and 
complexity of estimates and computations must be kept to 
a manageable number. 

H. CONCLUSION 

This chapter has presented a broad overview of cross- 
impact analysis, identifying its basic concepts and potential 
applications. Since there are a number of alternative 
approaches to implementing the methodology a set of evaluation 
criteria was developed to use in selecting an approach for 
operationalization. The next chapter will review the major 
cross-impact approaches, select one for demonstration, and 
extend it to deal with a crisis situation. 

Subsequent chapters will discuss the implementation 
procedure used, identify the problems encountered in opera- 
tionalizing the selected approach, and present the demon- 
stration results. In the final chapter, these results and 
the implementation experience will be used to evaluate the 
selected cross-impact approach's potential utility as a 
crisis decision-aid. 
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III. THE MAJOR CROSS-IMPACT APPROACHES 



A. INTRODUCTION 

This chapter will identify and evaluate the major appro- 
aches to cross-impact analysis using the evaluation criteria 
developed in the preceding chapter. Four major approaches 
will be described and discussed in terms of their basic 
assumptions, procedures and outputs. In addition, three 
ancillary approaches which were felt to be too complex for 
demonstration will be reviewed briefly. The four major 
approaches will then be compared using the evaluation cri- 
teria, and one will be selected for demonstration and evalua- 
tion as a possible crisis decision-aid. The selection pro- 
cess will also highlight possible problems in operationalizing 
the selected approach. 

Demonstrating the selected approach in a hypothetical 
policy relevant, crisis situation may require that it be 
extended to address specific problems inherent in such 
situations. These problems include short response time, 
and evaluating specific policy options. Possible extensions 
to cross-impact analysis will be developed once an approach 
has been selected. 

B. THE BASIC APPROACH (GORDON) 

The concept of cross-impact analysis was introduced by 
Gordon and Hayward in 1968 (Ref. 26]. As previously noted, 
it was a partial response to one of the Delphi criticisms 
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[Ref. 24] . Its first application was in a forecasting 
game , developed for the Kaiser Corporation, which tested 
resource allocation strategies [Ref. 24] . 

1 . Assumptions 

Cross-impact analysis is based on two assumptions. 

In response to the Delphi criticism, the developers hypothe- 
sized that most events are connected in some way. Further, 
they felt that it is possible to estimate these connections 
subjectively as to mode (enhancing or inhibiting) , strength 

(amount of influence) , and time lag (length of time required 

20 

for effect to be felt) [Ref. 26, p. 101-4]. 

Given that linkages existed and could be estimated, 
the developers next addressed how the impacts worked, or 
how one event's occurrence D m ,(E^) , changed a second event's 
D n , (E^) , occurrence probability. They assumed each event 
had some "individual" (inital) probability of occurring 
P n / (Pj_) / by a specific date. The change in probability 
P' ,(P'^), caused by another event's occurrence was assumed 
to be a function of mode, strength, and time lag. However, 
an event's failure to occur (non-occurrence) was assumed to 
have no effect on the remaining probabilities. 



2 0 

These relationships are more complex than indicated 
here. Gordon realized that the "connections are subtle 
and vary widely in strength, [sic] Composed of at least two 
major elements; probabilities of feasibility (P T ) and pursuit 
(P p ) •" [Ref. 24, p. 139] . 
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Stated mathematically, they assumed: [Ref. 26 



p. 104] 



where : 



P'i = f (P i ,M,S,t j ,t) 



f 

M 

S 

t 



function 
estimated Mode 



time in the future occurs 



= estimated strength 

= time in the future for which probabilities 
are being estimated 



The relationship between events was assumed to be quadratic 
[Ref. 26, p. 105] , 



P ' . = aP . + bP . + c 

i li 



By assuming known end conditions: 



when P^ = 0 



P. = 1 

i 



then PV = 0 



P' . = 1 



and if tj = t (no time lag) 



P . = P ' . 

l i 



And assuming the mode, strength, and time effects are 
linear [Ref. 26, p. 106]: 



a 
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where 



k = ±1 (Mode) 

S = number between 0 and 1 (smaller representing 
weaker effect) 

Therefore, in the inhibiting case, 

-1 < a < 0 



and enchancing 



0 < a < 1 



this produces: 



t-t. 2 t-t. 

P 'i = kS — t p i + tl-kS — - J ]P i 

2 . Procedure 

This equation was demonstrated in two applications, 
a historical context and a futures context. In these 
demonstrations, the estimates were made by the experimenters 
The historical problem examined the Minuteman deployment 
decision, the future problem examined transportation develop 
ments in light of technological advances and changing 
societal attitudes [Ref. 26] . 

In both cases the procedure began by specifying 
relevant events and developments to be examined. These 
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events were obtained through literature searches, expert 
interviews, and the analysts' own opinions. 

The next step was to estimate the initial proba- 
bilities and occurrence dates for each event. An event 
matrix was then constructed, and the potential interactions 
between each event and every other event were estimated. 

These estimates assessed both the interaction's mode 

and its strength. They were made by assuming one event was 

certain to occur, and then predicting its impact on each 

remaining event. These values ranged from -1.0 to +1.0, 

and made up the matrix body. Upon reviewing the individual 

event pair interactions certain "predecessor-successor" 

relationships were apparent to the experimenters. In the 

subsequent manipulation, events which had to be predecessors 

21 

were evaluated first. 

A computer was programmed with the quadratic 
relationship and the "Monte Carlo" procedure used to manipu- 
late the matrix. If an event was decided to occur, its 
impact was calculated from its cross-impacts with each 
other event using the quadratic equation, and the remaining 
events' initial probabilities were adjusted accordingly. 

If non-occurrence was decided then the remaining probabilities 



? 1 

In the matrix, these relationships were indicated by 
appending a second digit to the interaction values. (0 = 
immaterial, 1 = likely predecessor, 2 = necessary predecessor) 
[Ref. 26, p. 107] . 
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were not changed. The computer run was conducted one thou- 
sand times so that "revised" probabilities could be computed 
for each event based on its occurrence frequency [Ref. 26, 

p. 108] . 

3 . Analyses Performed 

Using this procedure, Gordon and Hayward demonstrated 
that four potentially useful analytical methods could be 
developed from the initial matrix. These were: (1) final 

probability rankings, (2) probability shifts, (3) tracing 
causal effects, and (4) scenario sensitivity analysis. 

Ranking by final probability produced a possible 
event sequence outcome scenario. In the example this was 
felt to be logically consistent (plausable) . However, the 
developers realized it was no more consistent than other 
possible scenarios. (Alternative scenarios could be obtained 
by recording the "decisions" made in each computer run using 
0 for non-occurrence and 1 for occurrence.) [Ref. 26, p. 108]. 

The ranking by probability shift (difference between 
and P' .) was used to identify the events most affected 
by the suspected interactions. In other words: "the item 

[event] which had the highest probability shift could be 
expected to be the one most influenced by the external 
events depicted by the remainder of the list." [Ref. 26, 
p. 108]. 

In their forecasting analysis, the developers 
attempted to use the matrix to trace through the reasons 
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for an event's resulting probability shift. For example 

they concluded that: 

Within the groups of items [events] 
describing future transportation 
modes, fast sub-surface trains 
showed the highest probability 
gain, suggesting that changing 
customer preferences, social 
customs, and technological 
innovations might favor this role. 

[Ref. 26, p. 113] . 

In essence, this is an elaborate way to state the 
probability shift conclusion, and later cross-impact developers 
would attempt to be more explicit in tracing causal effects. 

The developers also used their model to examine the 
potential effects of conscious policy decisions. Policy 
decisions were interpreted as decisions to allocate re- 
sources to making a particular event or set of events more 
likely. Such a decision was reflected by arbitrarily raising 
the initial probabilities for those events. The policy 
decision's results were then indicated by manipulating the 
matrix using the new initial values and comparing the 
effects produced, on the event sequence scenarios, with the 
initial run [Ref. 26, p. 115]. 

4 . Critique 

Gordon and Hayward recognized that their approach 
had several weaknesses. These would be expounded upon by 
later reviewers. The quadratic relationship had been selected 
due to its intuitive appeal and for convenience, however, 
the authors were not insistent that it be used. They 
suggested that other possible forms be examined. Gordon 
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and Hayward also recognized that there was little justifi- 
cation for assuming mode, strength, and time lag acted 
linearly . A logistic function was recommended as being more 
appropriate. The assumed non-effect of non-occurrence was 
felt to be an oversimplification which could be easily over- 
come. Finally, in future applications the developers pro- 
posed that estimates be obtained by a consensus— building 
technique such as Delphi [Ref. 26, p. 115]. 

One of Gordon and Hayward's critics, Enzer, criticized 

the quadratic adjustment assumption on the basis that it 

arbitrarily restricted the possible domain of change in 

P 1 ^ • The greatest absolute change in could occur only 

when P^ = 50%. The quadratic did not allow; 

. . . the occurrence of one event to 
have a large positive absolute 
impact on an event whose initial 
probability is very low, or a large 
negative absolute impact on an 
event whose initial probability 
is very high. [Ref. 35, p. 352]. 

He notes that although this appears to violate intuitive 

thinking, it did prevent PV from reaching unity or becoming 

negative . 

Enzer also explained the limitations produced by 
ignoring an event's non-occurrence. He felt that failing 
to consider non-occurrence leads to ambiguity in defining 
the initial probabilities. This is because any cross-impact 
effects which may have been implicitly included in estimating 
the initial probabilities are not removed. However, Enzer 
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noted that it may be conceptually difficult to estimate 
the effect of non-occurrence [Ref. 27, p. 59], 

In their estimation of the interrelationships, Enzer 
felt that Gordon and Hayward had not adequately defined the 
cross-impact factors [Ref. 27, p. 53J . This problem relates 
to how estimator responses should be interpreted, as dis- 
cussed in the preceding chapter. Two other critics, Mitchell 
and Tydeman, indicated this problem's implications when they 
observed that Gordon and Harward ' s "...procedure contains 
no mechanism to ensure that basic pairwise probability 
relationships remained intact." [Ref. 29, p. 134]. 

The basic approach had other problems associated 
with its use. Dalkey points out that relative likeli- 
hoods for individual event sequences (scenarios) could 
not be assessed directly. This was because only binary 
interactions were considered [Ref. 34]. 

To make scenario computation 
useful it will be necessary 
either to obtain estimates of 
higher order interactions or 
find a more logically correct 
assumption concerning them 
[Ref. 34 , p. 350] . 

Another problem noted by Mitchell and Tydemand was that the 
approach lacked an uncertainty measure with which to evaluate 
the expert's opinions [Ref. 29, p. 134]. Also, as discussed 
in Chapter Two, Johnson challenged the assumption that proba- 
bility shifts which were significant from a statistical stand- 
point could be assumed to be significant from a policy 
standpoint [Ref. 40] . 
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C. MODIFICATIONS TO THE BASIC APPROACH (ENZER) 

In two successive cross-impact analysis applications 
Enzer attempted to overcome some of the problems discussed 
above [Refs. 27 and 33] . In his earlier work Enzer retained 
the basic method's approach to quantifying event interactions, 
to using a quadratic manipulation, and to ignoring non- 
occurrence. The major innovation was that he used the 
Delphi technique to obtain his event set and subjective 
estimates. This allowed him to extend the analyses he 
could perform [Ref. 27]. 

By using the Delphi technique Enzer was able to retain 
the estimators' opinion spread for each specific relationship 
examined. Opinion spread is defined as the range between 
the upper quartile and lower quartile response values, or 
one standard deviation above and below the: mean value. With 
this information Enzer was able to conduct two analyses more 
systematically than Gordon and Hayward had. Enzer demon- 
strated that the mean response values could be used to iden- 
tify dominant and sensitive events before their probability 
shifts were determined. In addition, the estimator's opinion 
spread could be used to indicate estimator uncertainty 
concerning the potential interactions. 

An event's dominance was determined by summing the 
absolute values contained in a matrix row. Since these 
represented an event's impact on the other events, this 
measure was used to identify the events with the greatest 
overall impact. Event sensitivity was measured by alge- 
braically summing the column values. These values represented 
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the degree to which all other events influenced a single 
event's occurrence probability. This measure was used 
to identify the events which responded most readily to 
changes in the problem situation. 

The opinion spread in a row indicates the estimator's 
uncertainty regarding how an event's occurrence impacts 
the other events. The opinion spread in a column indicates 
the uncertainty concerning how an event is affected by 
the other events [Ref. 27, p. 55]. 

In addition to the standard estimates, Enzer elicited 
respondents' opinions concerning each event's importance. 

He suggested that these importance ratings, in conjunction 
with the above measures, can aid in identifying desirable 
events for resource allocation, and for selecting events 
to be used in sensitivity analysis. In this case, an important 
event which is insensitive to the others would require direct 
intervention, sensitive events might benefit from indirect 
action, and highly uncertain relationships would require 
closer analysis and evaluation before a decision could be 
made [Ref. 27, p. 57]. 

Although it introduced an uncertainty measure, and 
permitted more exact causal analysis Enzer 's approach 
suffered from many of the same problems associated with 
the Gordon and Hayward method. These recurring problems 
included: the use of the quadratic manipulation, ignoring 

non-occurrence, and determining how to interpret the 
respondent's estimates. Furthermore, the Delphi technique. 
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while extending the analysis , has some inherent problems . 
Fischer , et. al. cite studies which suggest that the Delphi 
technique (where individuals make estimates; receive anony- 
mous feedback about the estimates of other group members; 
and make additional estimates based on the feedback) is a 
poor way to elicit expert opinion. The studies reviewed in 
Fischer, et. al. indicate that the best procedure to elicit 
expert opinion uses a slightly different approach. The 
recommended procedure is to have the experts make individual 
estimates; then, bring the group together and allow the 
experts to discuss their estimates face to face; and finally, 
have the experts make a second set of individual estimates 
[Ref. 13, p. 55] . 

Enzer's second application introduced more fundamental 
changes to the Gordon and Hayward method. Enzer abandoned 
the quadratic manipulation, and introduced the concepts of 
"likelihood ratios" and "constant change of odds" to handle 
the procedure of manipulating estimates. In this approach, 
estimates were elicited as changes in probability and non- 
occurrence was considered. Enzer also attempted to resolve 
estimate inconsistencies [Ref. 33] . 

1 . Assumptions 

Using an event's importance rating, it was possible 
for Enzer to limit the number of events to be included in 
the analysis. Selecting the most important events avoided 
the necessity to specify the entire environment, thereby. 
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simplifying the analysis. However, this increased the 
complexity of the estimation process. 

Interrelationships were estimated in terms of the 
change in likelihood of one event (E^) produced by another 
event's (E ^ ) occurrence. This was defined as P(i|j). The 
effect of non-occurrence could also be estimated in the same 
way, and was defined as P(i|j). If this second estimate was 
too dififcult for respondents to provide it could be calcu- 
lated using a standard probability relationship [Ref. 33, 
p. 38] : 



P ( i ) = P ( j ) P (i | j ) + (1 - P ( j ) ) (Pi | j ) ) 

These values were then used to compute the likelihood 
(cross-impact) ratio for each event pair [Ref. 34, p. 350]: 

) = LzlUl x p (3 ID 

l ij P(j) 1 - P ( j I i) 

This became the factor by which the initial occurrence 
probabilities were multiplied when an event's occurrence 
or non-occurrence was decided. Using this ratio expanded 
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the domain of change possible for an event's initial 
22 

probability . 

To track the many changes in probability produced 
by the occurrence or non— occurrence of numerous events, 
Enzer introduced the concept of "constant change of odds". 
He stated: 

. . . the judgment concerning the 
impact on Event A of the occurr- 
ence of Event B is independent of 
the exact probability of occurr- 
ence of Event A, and that when- 
ever Event A is preceded by the 
occurrence of Event B the change 
in likelihood of occurrence of 
Event A is known from the initial 
input. [Ref. 33, p. 41]. 

This means that once the interaction between two events (A 
and B) has been estimated, its value is not changed if 
another event (C) occurs before B and changes event A's 
initial probability. As long as B occurs before A it will 
impact A's current probability to the same degree that it 
was believed to impact A's initial probability. 



22 

As Enzer explains in his earlier work: 

Changes in probability in the form of quanti- 
tative ratios opens the domain to the widest 
range of possibilities. Several of these 
are presented in Figure 6 in a form comparable 
with the previous figure. From Figure 6 it 
can be seen that an event whose initial proba- 
bility was 20% could, if impacted by an 
event that increased its likelihood by a 
factor of 20, be raised to 83%. When the 
quadratic is used, the maximum positive impact 
for an event with an initial probability of 
20% is to raise it to 36%. [Ref. 27, p. 61]. 
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In order to assure pairwise consistency was maintained 
in the estimates Enzer adopted the following axioms [Ref. 33, 
p. 37] : 

1. The initial probabilities of event pairs 
impose limits on the values of the impacted 
probabilities for that pair. 

2. For any event pair, specification of the 
initial probabilities and one set of 
impacted probabilities (occurrence or 
non-occurrence) fixes the remaining set. 

Thus, the relationship used to calculate non— occurrence could 
also be used to check for consistency. A simply boundary 
rule could then be applied to resolve inconsistencies. 

Pairwise consistency is required when dealing with proba- 
bility estimates to insure the calculated probabilities do 
not exceed unity or become negative. Such values would be 
meaningless in probabilistic terms. 

In applying this boundary rule Enzer assumed that the 
intention indicated by the respondents ' estimates were 
correct, but that their actual values were incorrect. The 
boundary rule then makes the least adjustment required to 
restore consistency. For example, if a value given for 
P(i j) in conjunction with those given for P(i) and P(j) pro- 
duced a computed value for P(i|j) which was <0 or >1, then 
it was inconsistent. A computer would set the value of 
P(i'j) = 1 or 0 as required, and recompute P(ijj) [Ref. 33, 
p. 38]. 

2 . Procedure 

Enzer' s approach followed the procedure developed by 
Gordon and Hayward. Candidate events for inclusion in the 
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analysis were obtained from a Delphi exercise. In order 
to limit the event set size, only those events considered 
most important (by subjective rating) were included. 

The respondents were then asked to estimate each 
event's initial probability of occurring by a specified 
date, considering an environment which might evolve from 
the present circumstances without conscious human interven- 
tion to change the course of events. They were then asked 
to rate each event's desirability. This rating included two 
subjective estimates; the desirability of an event to the 
environmental aspect being considered (+■ or -) , and its 
impact on that environmental aspect should the event occur 
(0 = weak to 10 = strong) [Ref. 33, p. 46]. This desira- 
bility rating seems to relate to subjectively assessing 
whether an event enhances or inhibits policy goals, or 
whether an event influences a situation in a direction 
desired by the decision unit. 

The respondents were next asked to assess the inter- 
relationships between events. They provided the new likeli- 
hood of occurrence for each event given each other event's 
occurrence (and non-occurrence if possible, otherwise this 
value was computed) . The estimates were then checked for 
consistency and revised as necessary using the boundary 
rule . 

Likelihood ratios were then computed to fill in 
the matrix, and the estimates manipulated as in the basic 
method, using a Monte Carlo simulation. 
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3 . Analyses Performed 



With this approach Enzer was able to perforin all 
the analyses permitted by the basic approach, including: 

1. determining even sensitivity and dominance 

2. scenario generation 

3. exploring causal relationships 

4. conducting scenario sensitivity analysis 

In addition he had obtained an indication of respon- 
dent uncertainty using opinion spread. Rating events by 
desirability allowed selected events to be included in 
the sensitivity analysis. It also permitted choosing the 
more desirable events as subjects for testing resource 
allocation strategies. 

Furthermore, desirability ratings provided criteria 
with which specific event sequence scenarios could be eval- 
uated. Using these criteria, it was possible to determine 
which scenarios were the most desirable, and which strate- 
gies produced them. 

4 . Critique 

This approach overcame several difficulties asso- 
ciated with the basic approach, specifically it 

1. considered non-occurrence 

2. eliminated the quadratic assumption' 

3. demonstrated that expert opinions could 
be used in analyzing complex situations 

4. provided an uncertainty measure for 
evaluating estimates 

Nevertheless, several problems still remain. 
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It was still difficult to identify significant 
probability shifts, although the desirability /impact and 
importance ratings had reduced the necessity for this 
determination. The interrelationships considered were 
still only binary, and even with desirability ratings it 
was still not possible to determine the most probable scen- 
arios. Finally, it was still unclear what the respondents 
meant when they provided interaction estimates. 

This last problem seems to have been, if anything, 

confounded by Enzer's attempt to introduce classical proba- 

tility relationships. As Mitchell and Tydeman observed: 

A fundamental difficulty with cross- 
impact analysis is to determine what 
respondents actually mean whn they 
answer the conditional probability 
questions normally posed. Results of 
studies currently in progress indicate 
that (1) participants are frequently 
confused and unsure of the interpretation 
of such questions, and (2) respondents 
often interpret the question in terms 
of time-dependent conditional probability 
statements , namely ' If event A occurred 
(first) , what is the probability of B 
occurring?' [Ref. 29, p. 133]. 

In addition, introducing conditional probability 
relationships stimulated the use of classical probability 
theory in determining estimate consistency. Also, Jackson 
and Lawton observe that the classical probability equation 
used to calculate non-occurrence and test consistency could 
also be used to derive half the matrix entries once the other 
half had been specified [Ref. 37, p. 265]. 
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Mitchell and Tydeman point out that, if the cross- 
impact estimates are interpreted as being time— dependent , 
Enzer's approach may not be valid. 



. . . the standard observation that 
P(A)P(B|A) = P (B) P (A|B) beomces, 
with the inclusion of time-dependence 



P (A) P (B | A f irst ) 



(=) 



P (B)p(A l B firs t ) 



In this restatement equality is not 
guaranteed - in fact, it would be 
pure coincidence other than when 
events A and B were independent. 

[Ref. 29, p. 135] . 

The consistency question led Enzer and Alter to 
address using Bayesian probability in cross-impact analysis 



They suggested that confusion in interpretation was caused 



because the early approaches [Ref. 39, p. 231]: 



1. Described future events with no real 
sense of time sequence. 

2. Used a data collection technique which 
blurred the distinction between correla- 
tion and causality. 



They concluded that conditional probabilities were 



not directly applicable to causal situations [Ref. 39, p. 237]. 



This conclusion also had been reached in 1972 by Turoff 
who proposed an alternative approach to cross-impact 



analysis [Ref. 36] . 



D. ASSUMED CONSISTENCY (TUROFF) APPROACH 

Turoff attempted to accomplish several goals with his 
approach. One was to develop a method which could determine 
the causal impact of potential events not specified in the 
event set. He also wished to establish some relationship 
between an event's occurrence likelihood and the effort 
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invested in either promoting or preventing the event. 

In presenting his approach Turoff acknowledged that 
because higher order interactions are difficult to esti- 
mate, "we in effect ignore terms of P or greater, hoping 
that the three- or four-way interactions are small." [Ref. 36, 
p. 326], Nevertheless, Turoff did attempt to derive a 
measure of the strength of these higher order interactions. 

1 . Assumptions 

Turoff 's approach was based primarily on the assump- 
tion that: 

Given a set of evnets which may or 
may not occur over an interval of 
time, we assume that an individual 
asked to estimate the probability 
of occurrence of each event will 
supply a 'consistent' set of 
estimates [Ref. 36, p. 317]. 

This assumption effectively removed the requirement to 

check estimate consistency. Instead estimates were obtained 

using an interactive computer program. In this, manner the 

respondent was shown the results produced by his estimates, 

and allowed to revise them at will until he felt the model 

they produced met his expectations. 

A second assumption in this approach is founded on 

information theory, which is used to calculate cross-impact 

factors (C. ■) from estimates used to derive values for the 

effect of occurrence (R^), and non-occurrence CS^j) for 

each event. The assumption was that "if a correlation 

exists between R. • and S. . is is such as to maximize the 

13 13 
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added information." [Ref. 36, p. 324]. The effect of 
this relationship can be shown simply. 



If e is defined as a quantity close to 0 then [Ref. 36, 
p. 324) , 



If P . = £ 

3 



1/2 



1-e 



then 





£ 



1 



1/e 



and P • 

l 



Re+S „ R+S 

1+e = 2 



R+eS 

l+£ 



R 



This behaves physically as one 
would desire, for if P. is close 
to one , then a very large change 
in R is necessary to make a small 
change in S. Conversely, if P. 
is close to zero, a very large^ 
change in S is necessary to pro- 
duce a small change in R. Also 
when P • = 1/2 the relative change 
in R and S is equal. [Ref. 36, p. 322]. 

Finally, a likelihood measure is used in expressing 
the probability that an event occurs, called its occurrence 
ratio (]>). This measure is similar to Enzer's likelihood 
ratio. Under assumed occurrence or non-occurrence it 
ranges from +«° to -«° [Ref. 36, p. 319] . The occurrence 
ratio is assumed to be directly proportional to a measure 
of effort, or [Ref. 36, p. 320]: 



<f>P ± “ 



E . 



i 



T 
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where 



>P . 
1 



= the occurrence ratio of event i. 



and 



V 



= the total amount of effort expended in 



either bringing about or preventing "i" 



Assuming that estimators are consistent, the assumed 
maximization of added information allows a difference equa- 
tion to be used for cases in which new information is 
received (i.e., an event's occurrence or non-occurrence 
becomes certain) [Ref. 36, p. 318]. Assuming certain 
occurrence (P ^ = 1), then, the approach defines ^ = P^, 
where R^^ is the probability of the i*"* 1 event given that j 
is certain to occur. Turoff then derives a cross-impact 
factor (occurrence ratio) for occurrence [Ref. 36, p. 321, 
(eq. 19) ] . 









Assuming certain non-occurrence (Pj = 0), then, the proba- 
bility of the i t * 1 event given that the event is certain 

to not occur is defined similarly, = P^. A cross-impact 

for non-occurrence is derived [Ref. 36, p. 321, (eq. 21)]. 





cjiS . . ] 
¥ 13 
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Combining these two factors produces [Ref. 36, p. 321, 
(eg. 22)], 






Obtaining values for all the ' s permits calculating 

Yj_/ a term assumed to be a function of the unknown variables 

not specified in the event set [Ref. 36, p. 321, (eg. 23)] 



n = » p i - . L c ik p k 



n 

l 

k^l 



By assuming the most favorable case for event i's occurrence, 

P.f, and the least favorable case, P. u , two y.'s can be 
x 1 1 1 

computed, y ^ and y^ U . This permits measuring explicitly 
the inaccuracy caused by ignoring higher order interactions 
[Ref. 37, p. 327, (eg. 45) ] . 



Y 



i 



f 



Y 



U 



i 



2. Procedure 

The Turoff procedure is essentially the same as 
the basic approach until the point at which scenarios are 
generated. As in the basic approach it is necessary to: 
specify the event set, obtain initial probability estimates, 
and obtain "conditional probability" estimates. These are 
used to calculate the cross-impact factors as discussed 
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previously . If there was no change in estimate from the 
initial probability then = 0. With this information 

it is possible to calculate the remaining measures described 
in the preceding section. 

Turoff criticized the Monte Carlo procedure as not 
producing a "consistent set of estimates" because the 
assumptions underlying it imply estimator inconsistency 
[Ref. 36, p. 309] . Rather than employing a random process, 
Turoff generates an event sequence scenario using the 
assumption that the events are time-independent in their 
occurrence . 

The first step under this assumption is to select 
the event whose initial probability is closest to 1 or 0. 

If P(i) is close to zero he assumes the event did not occur, 
if it is close to one it is assumed to have occurred. 

Based on this decision the probabilities for the remaining 
events are recalculated using the appropriate . This 
process is repeated using the events' 1 new probabilities 
until all events have been decided [Ref. 36, p. 331]. This 
produces a "unique" scenario. Other possible scenarios 
are produced by changing the initial probability estimates 
and repeating the above procedure. This is in effect, 
sensitivity analysis as done in the other two approaches. 

3 . Analyses Performed 

In presenting his approach, Turoff demonstrates only 
a sensitivity analysis. As in the basic approach, this is 
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accomplished by adjusting one or more of the initial proba- 
bilities and holding the remaining probabilities and the 
cross-impact factors constant. In addition to reflecting a 
conscious policy decision, this is the method whereby the 
estimator determines if the behavior of the model built from 
his responses is in accordance with his expectations. If 
he is not satisfied, it is at this point that he goes back 
and changes his initial and/or conditional estimates [Ref. 
36, p. 331] . 

Although other types of analysis were not demon- 
strated it seems most could be accomplished using this 
approach. Event dominance and sensitivity could be derived 
from Turoff's cross-impact matrix using a procedure similar 

to Enzer's. Since Turoff's cross-impacts are presented as 

23 

a type of correlation coefficient (± °° in range) , an 
indication of event desirability might be obtained through 
simple summing of the positive or negative effects. Desira- 
bility analysis would depend primarily on how the events 
were formulated. 

Turoff's matrix also includes a measure (Gamma) 
of the effects of events not specified in the event set 
(and higher order interactions) . This can be interpreted 
as an uncertainty measure. Estimator uncertainty could also 
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The use of correlation coefficients offers an alternate 
method of interpreting respondent's meanings when they 
answer the cross-impact questions. Its implications and 
possible uses will be explored in Chapter Five. 



be obtained by computing the range of opinion given by 
a group of estimators. 

4 . Critique 

Two major conceptual problems exist in Turoff's 
technique. The first of these arises from the assumption 
that estimators can be expected to provide consistent 
responses. Mitchell and Tydeman reject this assumption 
stating : 



[Turoff] argued that there is no need 
for usual traditional probability rel- 
ationships to hold and included a small 
example to support this assertion. In 
this example, he considered two time 
specific events and by undertaking 
usual conditional probability analysis 
concluded that P(A[b) and P(b|A) were 
not standard conditional probabilities. 

This is not surprising given the defin- 
tions which Turoff used; in fact, the 
finding is consistent with a time- 
dependent interpretation of conditional 
probability. The result does not support 
his claim that traditional probability 
relationships are irrelevant to cross- 
impact studies , and that violation of 
the usual probability relationships 
can be justified. The fact that 
people answer as though an ordering 
among events was implied seems to be a 
reason for explicitly recognizing this 
fact in the definition of the probability 
statement rather than ignoring it. [Ref. 29, 
p. 135]. 

It is difficult to conceptualize a situation in 
which an individual is able to operationalize all the 
relevant components of a complex problem, and quantify 
their interrelationships to produce a consistent set of 
probability statements defining his paradigm on demand. 
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Furthermore, assuming consistency removes the formal struc- 
ture provided the methodology by classical probability 
theory. Lacking an empirical base in the inexact sciences, 
it seems difficult to select an appropriate manipulation 
technique without some such formal theoretical basis. 

The second conceptual problem arises in the scenario 
generation process. By assuming time-independence Turoff 
produced one scenario, or event outcome sequence, but there 
seems to be little justification for assuming that it is 
unique. The assumption that the most probable event in a 
situation would occur first, or indeed occur at all, is 
debatable. This would be especially true in a situation 
characterized by surprise. This indicates that the 
procedure was established arbitrarily and that several 
others could have been devised which would have functioned 
equally well (e.g. , assuming the least probable event 
occurs and the most likely does not) . Without some method 
of identifying and ranking all possible outcomes, production 
of a single scenario seems pointless. 

A final criticism of this technique involves its 
procedure. Because respondnets had to reiterate their 
estimates until satisfied with the resulting coutput, an 
interactive computer system was required. To replicate 
this reiteration process for several respondents manually 
would require an excessive number of calculations be per- 
formed, and much time be expended in querying respondents. 
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The approach to be discussed in the next section seems to 
overcome these problems. 

E. CROSS-EVENT ANALYSIS (MITCHELL AND TYDEMAN) 

This approach was developed by Mitchell and Tydeman and 
presented in 1978 as a variation to Duperrin and Godet's 
SMIC-74, which is discussed in the following section. As 
has been shown, Mitchell and Tydeman felt it was inappro- 
priate to ignore time dependence when obtaining estimates. 
They felt that respondents normally interpret the questions 
posed for eliciting the conditional probabilities in a time- 
dependent manner. Mitchell and Tydeman attempted to use 
these time -dependent estimates to derive classical condi- 
tional probabilities [Ref. 39] . 

Mitchell and Tydeman also recognized that the elicited 
information could be inconsistent in terms of classical 
probability relationships. To minimize this problem they 
used average group responses which reduced the instances 
of pairwise inconsistency. In order to correct inconsis- 
tencies which did manifest themselves, they developed two 
fitting procedures, least squares and minimum standard devia- 
tions. These procedures will be reviewed below, the authors 
also suggested that Enzer's boundary rule could be used to 
resolve inconsistencies [Ref. 23, p. 18]. 

1 . Assumptions 

The major difference between this and the other 
approaches is that it uses causal time-dependent subjective 
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probability estimates to identify first order interrela- 
tionships among selected events, defining them as classi- 
cal probability relations. This approach uses a specific 
time horizon, rather than treating time as a random variable. 

Subjective estimates were obtained which completely 
identified the eight possible ordered outcomes permitted 
by any event pair. These outcomes can be identified explicitly 
for any event pair (i,j), and each has an associated unique 
probability (P^ through Pg) . These are [Ref. 29, p. 139]: 




Since these outcomes are exhaustive and mutually 
exclusive, certain conditions must apply [Ref. 39, p. 138]: 

P 1 + P 2 +P 3 +P 4 +P 5 +P 6 +P 7 +P 8 = 1 eq>1 

P i + P 2 +P 5 +P ? = P (i) eq. 2 

P, + P, + P c + P c = P (j ) eq. 3 

1 3 5 6 
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The authors feel that respondents can be expected 



to supply four time-dependent conditional probabilities 
[Ref. 39, p. 139] . 24 



P(i 


* -^first^ 


C 1 


= P 5 


r (p 5+ p 6 ) 


P(j 


1 d first^ 


= C 2 
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r (p 1 + p 2 ) 
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" C 3 
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t < P 7 + V 


P(j | 


d f irst^ 


= C 4 


= P 3 


f (P 3 +P 4> 



eq. 4 
eq. 5 
eq. 6 
eq . 7 



A fifth time-dependent relationship was also 
presented for estimation [Ref. 29, p. 139]. 



P(i firstl i and ^ occur) = C 5 = - (Pj_ + Pg ) eq. 8 

Solving these eight relationships as simultaneous 
equations, the developers demonstrate that the probabilities 
for an event pair's ordered outcomes can be calculated from 
the information provided by the respondents' estimates 
[Ref. 29, p. 140] . 



(C 3 C 4 ) - (Cjpp - <c 4 P i ) 



■ ra 



Q) - % c 4 ~ ^) c 3 
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If these last two equations are too difficult to 
estimate, the developers derive a method whereby they 
can be computed [Ref. 29, p. 141]. 
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where : 



a = 



““V + 1 



g = [ (1 - C 5 ) - C ± ] + C 5 , and 



Y = 



tc 5 <V + C 2 - (C 5 C 2> 



<C 2 C 1 ) 



Using these relationships Mitchell and Tydeman 
calculate the classical conditional probabilities for 
each event pair [Ref. 29, p. 140]. 

P (i I j.) = (P x +P 5 )/(P 1 +P 2 +P 5 +P ? ) 

P ( j | i) = (P x +P 5 )/(P X +P 3 +P 5 +P 6 ) 

As mentioned earlier, the developers derived a 
method to calculate non-occurrence if it could not be 
estimated. To calculate the conditional probabilities with- 
out estimating non-occurrence Mitchell and Tydeman adopt the 
following procedure. To obtain conditional probabilities 
one of the relationships needed is: 



A f- 



P(i and j) 



= P 



1 



+ P 



5 



or 



P(i and j) 



P(i and j first) + P(j and i first) 
P ^ i ^first^ P ^ first ^ + P ^l i first) 



P (i 



first 



) 



This can be obtained by estimating: 
or this can be approximated by: 



P (l first } 



and 



P(j first ) 



P(i^) = P(i^|i j) P(i) and similarly for P(j^) 

substuting : 



P(i and j) = C 5 C 2 P(i) + (l-c 5 )C 1 P(j) 



which is : 



P (i and j ) = P-^ + P^ + E 



where the error E is small for independent events, and more 
significant for highly correlated events (10-20%) [Ref. 29, 
p. 141]. 



e = (p 1 /p 1 +p 2 ) [ (p 1 p 7 -p 2 p 5 )/(p 1 +p 5 ) ] + (p 5 /p 5 +p 6 : 



(P 3 P 5- P 1 P 6» 

(P 1 +E V 
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Inconsistency is manifested when a negative value 

is calculated for any of the probabilities P, through P Q . 

1 8 

Standard fitting procedures may be used to resolve inconsis- 
tencies by hypothesizing that there exists a "true" or con- 
sistent value (*) for each respondent's estimate. This 
value is determined to be one which is "as close as possible" 
to the respondent's estimates and makes all P. , P Q > 0. 

The developers present two procedures which utilize this 
hypothesis: least squares, and minimum standard deviations 

[Ref. 29, p. 142] . 

The least squares procedure produces a minimization 
program "with a quadratic objective function and non-linear 
constraints" [Ref. 29, p. 142]. This program would not be 
easy to solve by hand. Minimizing the absolute standard 
deviations produces a moderately sized linear program with 
linear constraints [Ref. 29, p. 143-4]. In addition this 
program could be expanded to provide for overall consistency 
by including scenario probabilities and terms for higher 
order interactions [Ref. 29, p. 145], If this correction 
procedure proves too lengthy, the authors suggest Enzer's 
boundary rule could be used to resolve pairwise inconsis- 
tencies . 

2 . Procedure 

To obtain the events for analysis the authors used 
a procedure they termed "event assessment". For each 
candidate event, the respondents provide not only the 
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likelihood that the event occurs within a specified time 
horizon, but an estimate of its significance given that 
it occurs. An event impact score is then calculated as a 
weighted function of the product of probability and signifi- 
cance. In this way an analysis event set can be selected 
from the events having the highest impact scores, probabili- 
ties, or significance ratings [Ref. 23, p. 8], 

The next step is to determine the events ' time- 
dependent conditional probabilities by eliciting responses 
for questions structured to provide values for equations 
four through eight. These responses are aggregated and used 
to compute the ordered outcome probabilities P. through P 0 , 
which are used to compute the classical conditional proba- 
bilities. If any inconsistencies appear in this phase 
(any P^ through P g < 0) , they are eliminated using the 
boundary rule or fitting technique. 

The next step is to generate scenarios, or event 
sequence outcomes. For this procedure Mitchell and Tydeman 
used a variation of the Duperrin and Godet technique. This 
technique assumes that for an n nevent system there are 
r = 2 n states [scenarios) . Each state "E^,' r has a unique 
probability tt^ [Ref. 48, p. 305]. 

An event's theoretical probability .P* (i) is defined 
as its frequency of occurrence in all the possible states 
[Ref . 49 , p. 64] . 
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P* (i) 



for all i 



k=l 



e ik”k 



where 

' 

1 if event i forms part of E, 

9 ik = 

0 otherwise 

A similar definition can be applied to the theoretical 
conditional probabilities P*(i|j) [Ref. 49, p. 64], 



r 

P* (i | j ) = l t (ijk) ir./P* ( j ) for all i,j 

k=l K 



where 



t (ijk) 



1 when events i and j form part of 
0 otherwise 



and 



r 

P* (i | j ) = l s (ijk) it /I - P* (j) for all i,j 

k=l 



where 



s (ijk) 



1 when events i and j form part of E k 
0 otherwise 
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Recognizing standard probability constraints [Ref. 49, 
p. 65]: 



0 <_ P* (i) <_ l 

P*(i|j) = P* ( j | i) P* (i) = P*(ij) 

P*(i|j)P*(j) + P* (i|J)P*(J) = p*(i) 



produces : 



r 

l t(ijk)TT = P* (i j ) = P* (ji) 

k=l K 



and 



r 

P*(i|j)P*(j) = l s (i jk) it, = P*(i) - P*(ij) 

k=l X 



Therefore, given a consistent probability set, any set of 
values which satisfy: 



r 

P*(i) = l e. tt for i = 1,2, ...,n 

k=l lic K 



r 

P* (ij) = l t(ijk) for i = 1,2, .. . ,n-l; i < j <_ n 
k=l 



and 
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r 

I TT V = 1 ir, > 0 for all k 

k=l K k “ 



are possible solutions to the scenario probabilities 
[Ref. 49, p. 65] . 

Instead of the Duperrin and Godet quadratic objec- 
tive function, Mitchell and Tydeman solve a linear equation 
in terms of consistent probabilities to obtain a unique 
solution. This program is [Ref. 49, p. 65-6]: 

n 2 n — — 
minimize Z = £ [P(i|j)P(j) -P*(ij)] + £[P(i|j)P(j) -P*(i) 

ij ij 

+ P* (ij ) ] 2 



subject to: 



n n n 

l P*(i) - l l P* (ij ) 1 

i=l i-1 j=i+l 

P* (ij ) >_ 0 all j > i 

P* (i) <_ 1 all i 

% 

P* (i) > P*(ij) 



• all j > i 



P*(j) l P*(ij) 



1 



3 . Analyses Performed 

The cross-event approach uses a different method to 
determine event dominance and sensitivity. An event’s 
dominance may be estimated by summing , for all other events , 



91 



the absolute deviation of the conditional probability, 
given that the event occurred first, minus the event's 
original probability. This value can be standardized, 
or an "elasticity" measure adopted to reflect event likeli- 
hood. An event's sensitivity can be computed in a similar 
fashion [Ref. 23, p. 12]. 

Although it was not demonstrated, causal relation- 
ships could be traced as in the other approaches by filling 
in the cross-event matrix. This would be useful in an 
analysis Mitchell and Tydeman term "system impact analysis" 
which evaluates the possible event sequence scenarios' 
worth . 

System impact analysis used additional subjective 
ratings concerning an event sequence scenario's expected 
impact (magnitude and direction) on specified system com- 
ponents and the "expected consequences for system outcomes 
(performance measures)" (i.e., desirability) [Ref. 23, 
p. 14] . In this manner a different set of possible response 
actions may be identified for each scenario. 

The final analysis they suggest is "strategy 
evaluation". This procedure relates a set of feasible 
potential strategies (response actions as determined using 
system impact analysis) to each event sequence scenario 
(the most probable ones) , and uses a set of objective 
criteria (obtained from the decision-makers) to evaluate the 
strategies for each possible situation [Ref. 23, p. 15]. 
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4 . Critique 



A major advantage in this approach is that it 
eliminates the difficulty in interpreting estimator responses. 
Estimators are asked for time— dependent relationships which 
are then used to calculate classical conditional probabili- 
ties. Furthermore, if non-occurrence is too difficult for 
respondents to conceptualize, it can be calculated, 
introducing a small error. 

The capability to compute the conditional proba- 
bilities results in the further advantage of providing a 
logical background for the analysis. Classical probability 
relationships, it is suggested, can be used to insure con- 
sistency and resolve inconsistencies without requiring 
respondents to re-estimate. In addition, the procedure 
develops a method for calculating and ranking event 
sequence outcome probabilities. 

With the scenario ranking, introducing subjective 

estimates concerning system impact and objective measures 

for comparison extends the types of decision-aiding analysis 

which can be performed. With the Mitchell and Tydeman method 

it is apparently possible to address questions relating to 

the event sequence scenarios' desirability and strategy 

effectiveness. However, in system impact analysis: 

Problems still face the analyst, for 
example, eliciting an exhaustive objective 
set, ensuring mutual independence of 
objectives, devising procedures for 
measuring outcomes, determining consis- 
tent weightings for the objectives 
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and specifying procedures for aggre- 
gation. [Ref. 23, p. 15]. 

Tnese problems seem even more complex for decisions involving 
national objectives and foreign policy options. 

A final difficulty lies not in the conceptual 
realm, but in the method's implementation. Generating the 
scenario probabilities with the linear equation for a large 
event set would be very difficult without access to a 
computer. In order to replicate this process manually it 
appears necessary to severely limit the event set to be 
analyzed . 

F. OTHER APPROACHES 

Besides the approaches to cross-impact analysis dis- 
cussed so far, several additional techniques have been pro- 
posed. These techniques were concerned primarily with 
assessing the relative rankings among the event sequence 
scenarios. They also introduced alternative methods to 
insure estimate consistency, and to resolve inconsistencies. 

These techniques took significantly different approaches 
towards resolving these problems than the techniques just 
reviewed. However, each approach requires a computer pack- 
age. Thus they were evaluated as being inappropriate for 
demonstration and analysis in the present project. It was 
felt to be impractical to replicate their procedures for 
confirming consistency, establishing probability bounds 
and revising estimates using manual methods. However, these 
approaches have merit in presenting promising alternatives 
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to cross-impact analysis and will be discussed briefly. 

This discussion is offered in the interest of complete- 
ness, and to justify the conclusion to omit them from 
evaluation. 

1. SMIC-74 (Duperrin and Godet) 

Duperrin and Godet felt that cross-impact analysis 
could be used to construct and rank event sequence scenarios . 
In recognition of the fact that the estimators could be 
inconsistent in stating probabilities, they attempted to 
derive "theoretical" scenario probabilities which were 
implied by the estimator responses. Using those implied 
probabilities they attempted to revise the estimated 
responses to fit the theoretical scenario probabilities 
and produce a consistent set of estimate. The following 
diagram summarizes their procedure [Ref. 48, p. 306]. 



Figure 2 

THE SMIC-74 PROCESS 
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In order to accomplish this process Duperrin and 
Godet developed a minimization program to minimize the 
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difference between the estimated factors P (i | j ) resulting 
from the experts' responses and the theoretical factors, 
P*(i|j) and P* (i) , expressed in terms of even sequence scenario 
probabilities, , [ Ref . 48, p. 306]. The rationale behind 
this approach was covered in the Mitchell and Tydeman re- 
view. The Duperrin and Godet procedure utilized a minimi- 
zation program of quadratic form with linear constraints 
[Ref. 48, p. 306-7] . 

n r 7 
minimize Z = £ [P (i | j ) P ( j ) - £ t(ijk)iT ] 

ij k=l K 

n - r o 

+ £ [P (i | j ) P (j ) - l s (i j k) 7T, ] 

ij k=l K 



r 

subject to y ir. = 1 , tt, > 0, for all k 
k=l K K 

Obviously, solving this program would involve utilizing 
a large computer package. 

Furthermore, according to Mitchell and Tydeman the 
ranking produced is not unique "in that the quadratic 
programme has other possible solutions", and the "number of 
probability sets which satisfy the quadratic is potentially 
very large" [Ref. 49, p. 64], These conclusions led Mitchell 
and Tydeman to propose their approach discussed previously. 
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2 • Sequential Approach (Sarin) 



Sarin's approach is concerned primarily with event 
sequence scenario generation. Sarin derived a larget set 
of classical probability constraints which information 
elicited on initial and conditional probabilities must 
satisfy in order to be consistent (e.g., for a set of six 
events there are sixty— four conditions the initial proba- 
bilities alone must satisfy to be consistent.) [Ref. 38, 
p. 54-5], He then develops a set of linear equations with 
which to compute the 2 possible event sequence outcome 
probabilities and satisfy the constraints [Ref. 38, p. 59-60]. 

Sarin proposes that an interactive computer program 
be used in conducting the analysis. The program checks the 
estimator inputs for consistency, computes and supplies 
bounds for further estimates, and uses the consistent inputs 
to solve the linear equations producing the scenario proba- 
bilities [Ref. 38, p. 58]. This approach solves the consis- 
tency problem through providing respondents with immediate 
feedback. 

In this approach, it is possible to consider higher- 
order interactions with the computer's aid. Because these 
interactions are often so tightly bound it becomes unnecessary 
to estimate them [Ref. 38, p. 59]. Sarin also concludes that 
each discrete event influences only two scenarios, or event 
outcome sequences, and it is therefore possible to conduct 
sensitivity analysis through considering, and changing 
initial values for, only the events which influence the top 
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ranked scenarios. In this manner it is possible to estab- 
lish bounds on P (i) under which the scenario ranking does 
not change [Ref. 38, p. 60]. Similar bounds can be computed 
for the conditional probabilities, each of which influence 
only four scenarios. 

Once again, the need for an interactive computer 
program makes demonstrating this approach infeasible in the 
current study. Furthermore, no mention is made of conducting 
the other types of analysis (event dominance, sensitivity, 
and tracing causal effects) which would be necessary in a 
problem considering short term effects. Although such 
analyses could undoubtedly be accomplished, it would require 
making major modifications to the program. 

3. KISM (Kane) 

In 1972, Kane advanced a cross-impact procedure 
which dealt with developing trends rather than discrete 
events. His trend variables behaved in a logistic fashion 
(as Gordon proposed) . In this formulation the net effects 
of the cross-impact coefficients went to zero as the trends 
reached their maximum or minimum values [Ref. 47, p. 132], 

If these values are restricted to a range between one and 
zero, the impact of one trend on another is given by 
[Ref. 47, p. 133] : 

x^{t + At} = x^ (t) P i 



where 



or 



where 



P i (t) = 



. 1 + ¥ J/i^i -« 13 > x 3 



1 + 



At 



n 



l ( |a- • | + a. . ) x . 



j=l 



ID 



ID D 



(t) 



1 + At | sum of negative impacts on 
1 + At | sum of positive impacts on x^ 



a . 
ID 

x . 

l 



time 

the cross-impact value 
trend i 
trend j 



The relationship was modelled on a computer using the 
differential [Ref. 49, p. 133]. 



dx . 
x 

dt 



n 



l 

j=l 



a . . x . x . 
ID i D 



In x . 
x 



These developments can be modelled for any time increment 
and analyzed by varying the initial assumptions regarding the 
probabilities of the trends and the cross-impacts. Initial 
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trend and cross-impact values were obtained as in the 

2 5 

standard approaches [Ref. 47, p. 131]. 



25 

This approach's intuitive appeal is apparent from the 
computer output generated, which shows the interactions as 
they develop over time. A sample output for a transportation 
study is presented below along with the cross-impact matrix 
from which it was derived. The values in the matrix are a 
series of +'s and -'s which were converted directly to integer 
values. Another interesting point is that a trend can impact 
itself in this approach. Initial probabilities for the 
trends as established by the developer were [Ref. 47, p. 137] : 
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Lipinski and Tydeman have attempted recently to 
extend KISM to include a consideration of discrete events, 
non-constant trends, and their interactions [Ref. 50]. As 
with the other approaches discussed in this section this 
method suffers from the requirement that a large computer 
package be available for demonstration. 



G. MAJOR APPROACH EVALUATION 

Four primary approaches to cross-impact analysis have 
been identified and explained. 

The Basic Method — Attempted to elicit quantitative 

values for interactions among events and revised the 
initial probability estimates through a quadratic 
manipulation. The cross-impact concept used was 
incompletely defined, and subject to several inter- 
pretations . 

Enzer's Modification — Obtained estimates in terms of 

likelihood ratios, included non-occurrence, addressed 
pairwise consistency, and revised the conditional 
probability estimates. 

The Turoff Approach — Assumed estimator consistency, 
used inputs to derive correlation coefficients, 
and attempted to identify a unique scenario. 

Cross-Event Analysis — Derived conditional probability 
relationships from causal time-dependent estimates 
and used a linear program to identify and rank 
scenarios . 

Although the major problems associated with each of these 
approaches were identified in the discussion, selecting an 
approach for application cannot be made unless the approaches 
are systematically compared. The evaluation criteria developed 
in Chapter Two will be used for this purpose. Those criteria 
were: 
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1. Is the event scope adequate? 

2. Can events be rated on importance/desirability? 

3. What types of analysis are permitted? 

4. Is the approach computer adaptable? 

5. Are required estimates clearly formulated? 

6. Are estimates consistent, or can consistency 
be obtained? 

7. What manipulation algorithm is used? 

8. Is the procedure logical/ justifiable? 

9. How is consistency obtained? 

10. How are scenarios generated, can they be ranked? 

11. How many estimates are required? 

12. What types of computations must be performed? 

13. How many terms are involved? 

14 . How many calculations are required? 

1 . Evaluation 

The following table summarizes the findings discussed 
in the preceding sections in relation to these evaluation 
criteria. The four primary cross-impact approaches identi- 
fied form the column headings. Each entry in the table repre- 
sents a brief summary of how an approach was evaluated as to 
meeting the numbered criteria which form the row headings. 
Specific conclusions based on this evaluation will be pre- 
sented in the selection section of this chapter which follows 
Table One. 

A few of the criteria were omitted from the table 
because they were met equally well by all approaches. These 
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APPROACH EVALUATION 
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general criteria will be commented on below. In considering 
the last three criteria which addressed the required compu- 
tations' form and number (criteria 12, 13, and 14), two 
assumptions were made: (1) The types of computations and 

number of terms (criteria 12 and 13) include all possible 
computations which could be necessary (including computing 
cross-impacts and resolving inconsistencies) in demonstrating 
the approach, except for those involved in aggregating esti- 
mates to produce an average value. These computations 
would not be approach dependent. (2) The number of computa- 
tions (criterion 14) does not include computations required 
in either aggregation nor resolving inconsistencies, as the 
latter depends on the knowledge of the respondents . 

The general criteria were found to be: 

a. Event Scope (1) 

Event formulation in all the methods can cover 
any subject or combination of subjects. The only restric- 
tions concern the linkages between events, and the way in 
which events are stated. These restrictions apply equally 
to all approaches. 

b. Event Rating (2) 

Although each approach did not rate events for 
selection for analysis, the procedures proposed by Enzer and 
by Mitchell and Tydeman are flexible enough to be applied in 
any cross-impact analysis. Thus once a technique is selected 
it is necessary only to select a method to evalute the events. 
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a.nd explain the criteria for subjective estimates to the 
respondents . 

c. Computer Adaptability (4) 

Each method reviewed acknowledged that a computer 
would be necessary in order to deal with a large event set. 
Turoff's approach required that an interactive computer be 
available to query respondents, and present them with the 
results produced by their estimates. Enzer proposed that a 
computer be used to provide probability bounds based on 
estimator responses for each event pair. It appears that 
all necessary calculations could be easily programmed, and 
that a software package could be developed to elicit responses 
for any type of event. Therefore, it is felt that all the 
approaches may be considered computer adaptable. 

The evaluation of each approach in relation to 
the remaining criteria is presented in Table One. This 
evaluation highlights the major differences between each 
approach . 

2 . Selection 

As the discussion below indicates, of the four major 
cross-impact approaches evaluated, the Turoff method seems 
to be the least promising for demonstration in this study. 

The basic (Gordon) method was developmental and substantially 
extended by Enzer. Enzer 's approach, however, involved 
difficulties in relating respondents ' estimtes (which could 
be interpreted as providing causal time-dependent probabilities) 
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with classical probability theory. This shortcoming would 
make it difficult to explain the technique to respondents. 

In contrast, the Mitchell and Tydeman approach develops 
procedures to calculate classical conditional probabilities 
from the apparently time-dependent estimates, and provides 
a scenario ranking algorithm. For these reasons it was 
selected for demonstration in this study. The following 
discussion will present the rationale behind these conclusions, 
and indicate anticipated problems in implementing the selected 
approach . 

Due to its assumption that estimators are consis- 
tent, the Turoff approach requires that they be allowed to 
revise their estimates until satisfied with the model pro- 
duced. This requirement, as mentioned previously, would be 
hard to meet in a manual demonstration. The assumption of 
consistency cannot be validated because the cross-impacts 
are calculated in terms of correlation coefficients (ranging, 
however, from +°° to -°°) , which do not reveal inherent incon- 
sistencies as do probabilities when they exceed unity or 
become negative. 

The Turoff approach is further limited by its time- 
independence assumption for scenario generation. This pro- 
duces only one scenario for each set of estimates, and there 
is no reasonable basis for assuming that it is either a 
unique or most probable scenario. Finally, the time- 
independence assumption would be unworkable in an analysis 
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attempting to examine the effects produced by discrete 
response options which might be implemented in reacting to 
a developing crisis situation. The rationale and structure 
of this type of analysis will be developed more fully in the 
extension section of this chapter which follows. 

The basic cross-impact approach, developed by Gordon 
and Hayward, while important in introducing the technique's 
basic concepts and potential applications was primarily a 
developmental approach. As such it suffered from several 
conceptual shortcomings which were eliminated in later 
approaches. These shortcomings were reviewed extensively 
elsewhere in this study, and stem from the use of the quadratic 
manipulation technique, ignoring non-occurrence, and incom- 
plete explanation of the cross-impact factors. Because this 
approach was greatly modified by Enzer it was decided to 
eliminate it from further consideration for selection. 

Enzer' s modified approach dealt with many of the 
problems inherent in the basic method. He developed an 
uncertainty measure, addressed non-occurrence and consistency, 
and eliminated the quadratic manipulation. However, in 
developing these improvements he failed to solve the problem 
of how to interpret estimator responses. Enzer recognized 
that causal relationships were being elicited, but used 
classical probability relationships to compute non-occurrence 
and resolve inconsistencies. The cross-impacts were defined 
as likelihood ratios denoting an event's change in probability 
given the occurrence of another event. This change was 
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denoted by the conditional probability relationship P(i|j), 
however , the actual relationship seems to involve time- 
dependence. Therefore, this approach seems to be limited by 
the incomplete incorporation of two types of probabilistic 
relationships. In order to demonstrate a technique it will 
be necessary to explain its rationale to the selected study 
group. This requirement makes the modified approach unsuit- 
able for demonstration because of the difficulty in explain- 
ing these fine distinctions and their implications to respon- 



dents unaccustomed to providing this type of information. 

Cross-event analysis assumes that respondents normally 
think in terms of time-dependent probabilities, and uses 
these estimates to calculate the conditional probabilities 
from the relationships implied by the estimator's responses. 
Because this approach seeks to explicitly identify quanti- 
tative values for all possible outcomes of each event pair, 
a greater number of subjective estimates is required. How- 
ever, the questions used to obtain these estimates are formu- 
lated in a manner which appears to be more closely akin to 
suspected individual thought processes ) . 

Since these estimates are used to compute the proba- 
bilities of the eight possible ordered outcomes and then to 
calculate the classical conditional probabilities, more com- 
putational steps are involved in using this procedure. These 
formulae are relatively basic and can be easily programmed 
on a hand-held calculator. This extra burden is, therefore, 
not considered excessive in light of the advantages gained. 
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Using the time— dependent causal estimates, pairwise 
probabilistic consistency is automatically checked when com- 
puting the probabilities of the ordered outcomes. By 
specifying values for these outcomes it is possible to derive 
a logical transition to the classical conditional proba- 
bilities. This avoids the problem of applying Bayesian proba- 
bility relationships to causal estimates, and eliminates the 
need to adopt less supportable assumptions - such as estimator 
consistency. At the same time, because it is possible to 
use the classical probability relationships, the Mitchell- 
Tydeman method is given a firm methodological footing. 

Finally, Mitchell and Tydeman ' s cross-event approach 
provides an algorithm for computing scenario probabilities. 
This capability is necessary if decision-makers are to be 
provided a means of assessing possible courses of action in 
responding to ongoing developments. Scenario probabilities 
cannot be obtained under the Turoff method's assumption of 
time-independence, and could only be derived from the Enzer 
method by conducting an excessive number of Monte Carlo runs 
(exceeding the number of possible event sequences) . 

For these reasons, the Mitchell and Tydeman cross- 
event analysis method was selected for demonstration and 
evaluation in this study. Because the purpose of the demon- 
stration is to evaluate cross-impact analysis' potential as 
a decision aid in crisis situations, several modifications 
to the approach were considered necessary. These modifica- 
tions, presented below, were intended to extend the method 
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in order to evaluate policy options for response in a 
developing situation. 

H. EXTENSION 

This section will discuss several methodological 
alterations through which cross-event analysis can be 
extended as a decision-aid for use in crisis situations. 

Four aspects of the methodology will be dealt with. These 
are: (1) the time span of analysis, (2) identifying policy 

options for inclusion in the event set, (3) the method of 
analysis, and (4) scenario generation. 

1 . Time Span 

Crisis situations normally occur without adequate 
warning and allow a short time for response. As Enzer and 
Alter pointed out, the time span usually considered in a 
cross-impact analysis is determined by the amount of time 
required for an event's impact to be felt by another event 
[Ref. 40, p. 233]. Since the technique being explored here 
is intended to evaluate response options, it is felt that 
this time span should be kept relatively short. 

In a crisis situation, it seems reasonable to assume 
that most response options would be designed to have an 
immediate effect in altering the situation in some desired 
direction. For this reason, it appears desirable to limit 
the analysis to a consideration of the two weeks or so follow- 
ing the identification of a crisis. As there is almost no 
time lag involved, the event prediction need only involve 
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occurrence and non-occurrence, rather than exact dates. This 
is a level one analysis described in Johnson [Ref. 40] , 
which predicts only those events which occur by a specific 
date. In providing initial estimates, respondents will not 
be asked for occurrence dates, in providing causal time- 
dependent estimates any event may be considered to occur 
before any other. 

In the analysis, the sequence in which events occur 
will be revealed through scenario generation. As will be 
shown below, this will aid in evaluating likely scenarios for 
plausability . This assumption would not be valid if the 
effects of policy options were to be examined in their full 
context. As has been noted, careful evaluation would require 
an assessment of a response option's long-term effect, not 
only on the crisis situation. In a "real world" application 
the short-term and long-term analyses could be conducted 
simultaneously to identify a possible response to control the 
crisis and to evaluate its potential future effects. Such 
an approach was not adopted here because of this study's 
developmental and exploratory character. 

2 . Identifying Policy Options 

The normal procedure in cross-impact analysis has 
been to evaluate the effects produced by implementing policy 
options through arbitrarily increasing or decreasing selected 
events' initial probabilities. This seems to be contrary to 
a basic goal of the approach, which is to identify an event's 
unanticipated effects. Furthermore, Gordon and Hayward 
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hypothesized that an event normally has a predecessor or 
precedes another event [Ref. 26] . It was the purpose of the 
technique to systematically assess these interactions. In 
the normal sensitivity analysis, a policy option is excluded 
from the matrix, and assumed to have only the effect intended. 

In the Mitchell and Tydeman approach, subjective 
assessments are made of the impact that specific strategies 
have on the various scenarios. This requires a prior "Sys- 
tem Impact Analysis" to assess how the occurrence of a 
scenario impacts identifiable system attributes, and the 
development of "objective criteria" to evaluate policy 
strategies. It has been pointed out that the validity of 
this process would be tenuous for problems involving national 
objectives. Furthermore, it increases the number and com- 
plexity of the subjective estimates required. 

In an effort to simplify the process of assessing the 
impact of policy options on situational outcomes, this 
study will attempt to identify specific policy responses 
available for dealing with the hypothetical situation and 
include them in the original cross-event matrix. These 
response options will be identified and assessed as any other 
event which might develop from the given situation under 
normal circumstances. Using this approach will eliminate 
the need for "System Impact Analysis". Normal sensitivity 
analysis will still be conducted to determine response option 
flexibility in the event that available information was mis- 
leading and/or the problem components were evaluated incorrectly 
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by the respondents. Including response options in the event 
set also permits other extensions. 

3 . Method of Analysis 

One possible extension involves the type of analysis 
performed by evaluating the causal relationships contained 
in the cross-event matrix. If the events representing policy 
options are separated from the events representing situation 
developments a four-quadrant matrix is formed as represented 
below. 



FIGURE THREE 

FOUR-QUADRANT CROSS -EVENT MATRIX 




(note: the diagonal cells can still contain the initial 

probabilities) 



Because the subjective estimates used to obtain the 
conditional probabilities are in a causal time-dependent 
form, the following interpretations might be given to each 
of the four quadrants. 
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I* (°|d) Represents the situation as it might 
develop assuming no corrective action is taken. 
This depicts the results of allowing a given 
crisis to continue. 

II. ( 0 | D ) Represents how the occurrence of given 
developments effect the probabilities of various 
response options . The occurrence of certain 
developments may make a response option more 
attractive. (It is felt that at this point not 
enough information is available to use this 
interaction to predict which response option 
may be selected, only whether or not the 
respondents consider it more attractive.) 

III. (D|0) — Represents the effect on the situation 
produced by the occurrence of each response 
option. This would be the primary quadrant 
with which to evaluate the response options. 

IV. (o|o) — Indicates the interaction among 

possible response options. This possibly could 
be used to trace the impacts of a mixed 
strategy, or to evaluate the effects of imple- 
menting more than one response. 

As in the traditional approaches, each of these 
quadrants could be evaluated to determine event dominance, 
sensitivity, and estimator uncertainty. Other forms of 
analysis would remain essentially the same. These inter- 
pretations can only be hypothesized at this time. It will 
require experimental data to determine their applicability. 
Therefore, this type of analysis will be attempted using the 
results from the demonstration exercise. 

4. Scenario Generation 



Since specific policy options are to be included 
in the event set, it is questionable as to whether they 
should be included in the event sequence scenario generation. 
The occurrence or non-occurrence of a policy option is not 
determined solely by the situation developments but also by 
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the decision maker's perceptions and the evaluations they 
make. The policy options are explicitly included in order 
to facilitate more complete evaluation. Based on this 
information, the decision-makers would then select the 
option (s) to be implemented (i.e., decide occurrence) . 

It might also be argued that by including one, or 
all, options in event sequence scenario generation it would 
be possible to determine which option (s) should be implemented 
(based on their occurrence or non-occurrence in the most 
probable scenario). In other words, generating and ranking 
scenarios with N options (N = 1,2, 3, . . . ,n) , and evaluating 
these scenarios for desirability would indicate which 
options produced the most desirable outcomes. 

However, in a short term situation, the specific 
event sequence scenario may not be so important. Accurate 
information is limited, and the unexpected could always 
happen (including events not specified in the set) . There- 
fore, scenario generation intended to provide specific predic- 
tions would be of questionable value unless the problem was 
clearly understood by the estimators. In view of these 
considerations, this demonstration will only include the 
situation developments in event sequence scenario generation. 

I. CONCLUSIONS 

This chapter described four major approaches to conducting 
cross-impact analysis, including their basic assumptions, 
computational procedures, and types of analysis performed. 
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Based on this discussion, and using consistent evaluation 
criteria, the approaches were compared and the Mitchell- 
Tydeman cross-event analysis was selected for us in this 
study . 

Cross-event analysis was selected because, although it 
required obtaining more numerous subjective estimates, its 
methodology and underlying assumptions seemed more systematic 
and thorough as well as intuitively pleasing. In addition, 
the calculations required in the procedure appeared to be 
simple enough to perform without using a computer. The 
Mitchell and Tydeman approach was extended for application 
in a crisis situation. This extension involved: limiting the 

time span considered in the analysis, including policy options 
as discrete events for analysis in the matrix rather than 
through sensitivity analysis alone, and restricting scenario 
generation to include only situational developments while 
excluding policy options. 

As indicated in the discussion of the major approaches, 
several problems may still be encountered in operationalizing 
this approach. Some of these problems are inherent in 
conceptualizing the cross-impact technique itself and can- 
not, therefore, be resolved until data have been collected 
and analyzed. Such problems include: how estimators can 

best quantify the interrelationships being sought (proba- 
bilistically or as correlations) , how to interpret estimator 
responses (as classical probabilities, causal time -dependent 
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probabilities, or as correlation coefficients), and how 
inconsistencies should be treated (ignored, boundary rules, 
or fitting methods) . The selected technique assumes respon- 
dents can think in probabilistic terms, indeed, in terms of 
causal time-dependent probabilities, and that inconsistencies 
are best resolved using fitting procedures. 

Other potential problems arise from dealing with subjec- 
tive estimates, and from the methodology used in the selected 
approach. These include how best to aggregate estimates, 
and how to resolve inconsistencies. Both of these depend on 
the form in which the estimates are elicited and the actual 
values obtained. Therefore, these problems will be examined 
in the context of the selected technique's operationalization. 

The next chapter will discuss the actual implementation 
process and present the demonstration results. The imple- 
mentation process will describe how the demonstration was 
structured, and the problems encountered in using the Mitchell 
and Tydeman approach. The demonstration results will be used 
to evalute the accuracy of the approach's assumptions, and 
its potential utility as a crisis decision-aid. 
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VI. OPERATIONAL! ZATION 



A. INTRODUCTION 

Employing cross-impact analysis in an experimental appli- 
cation requires developing and following a logical implemen- 
tation process. Due to the approach's information require- 
ments and the type of problem proposed, four preliminary 
steps are necessary for implementation. The first task is 
to identify a study group to act as expert respondents. Next, 
a problem environment for analysis has to be constructed. 

Then, this problem environment, and the approach's purpose, 
methodology, and information requirements have to be explained 
to the study group. Finally, a procedure to elicit the group's 
subjective estimtes has to be devised. 

Once the preliminary background has been laid, actual 
implementation requires collecting and structuring the res- 
spondents ' estimates into the format required by the cross- 
event approach, and conducting the analysis. This process 
includes : identifying candidate events and selecting from 

these the analysis event set, and aggregating individual esti- 
mates into group response values to provide the data for 
replicating the analysis performed by Mitchell and Tydeman 
[Ref. 29]. 26 



2 6 

The purpose behind using group values is two-fold. 
First, it should reduce instances of pairwise inconsistency 
[Ref. 29]. Secondly, as proposed by Enzer, the respondents' 
opinion spread could serve as an estimator uncertainty 
indicator [Ref. 27, p. 55]. 
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To replicate the cross-event analysis, as extended in 
the previous chapter, it is necessary to perform two calcu- 
lation steps. The first step is to compute the event pair 
ordered outcome probabilities (P^ through P g ) . These values 
are then used in the second step which calculates the con- 
ditional probabilities implied by the respondents' estimates. 
The conditional probabilities can then be arranged into a 
cross-event matrix and scenario generation undertaken. These 
tools are then used to perform the problem analysis. 

This chapter describes the implementation design developed 
for this study. It discusses how the actual operationaliza- 
tion evolved by reviewing the successes and failures encoun- 
tered in following the designed implementation. Finally, the 
data collected are presented and analyzed. 

B. IMPLEMENTATION DESIGN 

The entire operationalization process was designed to 
follow the demonstration described by Mitchell and Tydeman 
[Ref. 29, p. 145-147]. Their "event assessment" procedure 
was used to select an analysis event set [Ref. 23, p. 6-9], 
the forms for obtaining subjective estimates were taken from 
the examples contained in Ref. 29, and the equations used 
in all calculations were from the derivations presented in 
the same source. 

The procedure used to demonstrate cross-event analysis in 
a hypothetical situation was constructed to follow five phases. 
Each phase's purpose and structure will be discussed be_ow. 
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1. Scenario Construction 



To establish the problem environment for analysis a 
hypothetical problem scenario was developed. Several con- 
siderations influenced this development. Because the problem 
was intended to represent an emerging crisis situation, the 
scenario had to threaten identifiable national interests, 
suggest potential policy options which could be identified 
as responses, and involve recognized actors. This last con- 
sideration was adopted for two reasons. Using real actors 
would provide an actual basis for the respondents 1 percep- 
tions rather than having to structure perceptions by speci- 
fying a host of hypothetical historical, social, political, 
economic, etc. data in the scenario. Also, using real 
actors was expected to make estimation and analysis more 
meaningful to the respondents. 

To satisfy the above requirements and involve as much 
current information as possible in the problem environment, 
the scenario was based on developments in South Asia. A 
brief justification for this decision will be presented in 
the next section which describes the actual scenario pre- 
sented to the respondents. 

The scenario was presented in four sections to allow 
as complete a development as possible. Those sections were: 

(1) Background — described the primary actors, their internal 
situations and external relationships. (2) Event Sequence — 
listed the hypothetical moves and counter-moves taken by the 
various actors. (3) Resulting Situation — set forth the 
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actual problem (crisis) to be analyzed and responded to. 

(4) U.S. Concerns described what appeared to be the 
significant U.S. interests in the area. The scenario 
presentation was conducted as the second segment of the 
design implementation's explanation phase. 

2 . Explanation 

This phase described the experiment's purpose and 
methodology to the study group. This explanation was in- 
tended to help the respondents understand their role in the 
experiment. It included: The rationale behind using sub- 

jective estimates, the information the respondents would 
be asked to provide, the information format, how the infor- 
mation would be analyzed, and answered any questions which 
arose. In this manner it was hoped to fully involve the 
respondents in the project and thereby produce more care- 
fully considered responses, while demonstrating to the 
respondents the approach's potential as a learning tool. 

The explanation phase involved two segments. The 
first described cross-impact theory, its assumptions, 
methodology and proposed applications. Once the basic 
concepts and their use was understood, the Mitchell and 
Tydeman approach was described to the study group. This 
presentation highlighted the functions the respondents 
would be required to perform. 

The second explanation segment presented and dis- 
cussed the hypothetical scenario. This was presented in 
four sections, as previously described, in order that the 
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P art -'- c -'-P ants could fully understand the problem elements 
involved. In an actual application this segment would be 
optional, as the experts" could be expected to be familiar 
with the problem environment. Once the respondents under- 
stood the technique and hypothetical problem the third 
implementation phase was begun. 

3 . Elicit Estimates 

This phase was designed to obtain the subjective 
estimates used in the problem analysis. Two estimation 
procedures were developed for this process. The first 
used a "brainstorming" technique to elicit a candidate 
event set. Immediately after the scenario discussion the 
respondents were asked to suggest developments which might 
occur resulting from the proposed situation. All subse- 
quent estimates were obtained using an individual response 
procedure . 

Once the candidate event set was established the 
respondents were asked to individually rate each event's 
initial occurrence probability and significance. This 
information was used to conduct an "event assessment" and 
select a manageable event set for analysis. As discussed 
previously, this procedure was adopted to reduce both the 
number of estimates and computations required by concen- 
trating on only the "important" events. Individual esti- 
mates were then obtained for the causal time-dependent 
probability relationships (C^ through C,.) existing in each 
event pair. These data were intended for use in the fourth 
implementation phase. 



4 . Conduct Analysis 



The fourth phase was designed to structure the 
respondents' estimates into the cross-event format and 
analyze the resulting output. Individual responses were 
aggregated into group values and used to compute the 
"cros s— impacts " from the formulae developed by Mitchell 
and Tydeman. This procedure computed the ordered event 
pair outcome probabilities (P^ through Pg) and the resulting 
conditional probabilities for each event pair. At the same 
time, pair-wise consistency could be checked and resolved 
if necessary. Because group values were used, instances 
of inconsistency were expected to be rare. 

Given consistent estimates a cross-event matrix 
would be constructed and scenarios generated. These data 
would then be analyzed as described in Chpater Three. The 
analyses to be performed included: For events — determining 

dominance, sensitivity, signficance, and uncertainty. For 
interrelationships — conducting causal tracings through the 
event matrix (as discussed in the extension section of Chap- 
ter Three) , evaluating response options using the matrix 
and scenarios, performing scenario and sensitivity analysis. 
The implementation design included presentation of the 
analysis results as its final section. 

5 . Presentation of Results 

This phase was designed to organize and present the 
analysis results for evaluation. These results were pre- 
sented to the study group to close the feedback loop and 
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complement the learning experience. This phase also 
involved the participants in evaluating the cross— impact 
technique. 

Respondents were asked to suggest possible reasons 
for the results obtained, and to assess whether or not 
the information derived from their estimates met their 
expectations. They were also asked to comment on their 
impressions concerning the technique's procedures and poten- 
tial utility. This information, and the analysis results, 
were used in evaluating cross-event analysis as a possible 
crisis decision-aiding methodology. The respondents' 
feedback and the actual implementation experience were used 
to evaluate the procedure's ease of application, to iden- 
tify possible problems and to recommend improvements. The 
following section recounts how the designed implementation 
actually evolved, and identifies the successes and problems 
encountered in operationalizing cross-event analysis. 

C . ACTUAL IMPLEMENTATION 

The preliminary implementation steps were relatively 
easy to accomplish. The study group utilized was a Data 
Analysis for Naval Intelligence class being conducted 
(Spring Quarter, 1979) at the Naval Postgraduate School, 
Monterey, California. The class provided ten respondents 
(nine Naval officers and a civilian Professor) . The offi- 
cers had various types of operational experience in the 
U.S. Navy, and were undertaking graduate studies in the 
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school's National Security Affairs Department. Thus, 
their background was felt to be adequate for demonstration 
purposes. All respondents had some exposure to probability 
and statistics, organizational management and decision-making, 
and various topics in National Security and Intelligence. 

In addition, most had kept abreast of ongoing developments 
in South Asia. These characteristics simplified the explana- 
tion process as no really "new" material had to be covered. 

The explanation phase previously outlined consisted 
essentially of orally summarizing Chapter Two and the cross- 
event analysis section in Chapter Three of this study. The 
explanation also reviewed the procedure designed to elicit 
the group's subjective opinions. In the second explanation 
phase, the study group was led through the scenario developed 
for this demonstration. The problem scenario presented to 
the respondents will be described below, followed by des- 
criptions of the remaining implementation phases. 

1. Scenario Development 

Developments in South Asia and around the Persian 
Gulf have become a focal point for U.S. diplomatic, economic, 
and military interests. This increasing interest has been 
well documented by noted political scientists and historians. 
These authors have detailed increasing U.S. interest in. 
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and commitment to, the region resulting from numerous 

27 

interrelated factors. 

Some of the primary factors influencing U.S. 

interests in the area include, increasing U.S. dependence 

on imported oil and the tenuous supply links through which 

2 8 

it is obtained. Another factor is the increased Soviet 

influence in South Asia, their increasing naval presence in 

2 9 

the Indian Ocean, and the purpose behind this presence. 

A third concern is the continuing tension in the region 
which seems to keep the relationships in the area in a 



For example, Nadav Safran discusses the U.S. involve- 
ment growing out of the Middle East peace process [Ref. 51] . 
Richard Ullman has examined U.S. policy options in the 
area in light of continuing tensions and outside pressures 
[Ref. 52] . Basic considerations for U.S. policy in this 
area were presented by Campbell and L'enczowski [Refs. 53 
and 54] . Inter-regional relationships and super-power roles 
are examined by Vali [Ref. 55] , while Wall has collected 
papers addressing relevant strategic concerns [Ref. 56] . 

Many of the interests and concerns developed by these 
authors are reflected in official U.S. Government docu- 
ments such as the "Department of Defense Authorization 
for Appropriations for FY 79" policy statements [Ref. 57]. 

2 8 

The oil question is explored by analysts such as 
Lenczowski and Anthony [Refs. 58 and 59]. Its future 
implications are examined in a CIA forecast [Ref. 60] , 
which indicates that the situation is unlikely to improve. 

29 • • 

Increased Soviet influence and military presence is 

discussed by Lenczowski [Ref. 61], Hurewitz [Ref. 62], 

Tahtinen [Ref. 63J , and McConnell [Ref. 64]. The dispute 

over the intentions behind this presence has produced a 

resultant U.S. policy debate over its military presence 

reviewed in Daniel [Ref. 65]. 
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The complexities of these 



state of flux and instability. 3 ^ 1 
problems are indicated by several authors. Although a complete 
development of these and associated factors is beyond the 
scope of this study, that the region's importance is an area 
for continued policy analysis should be self-evident. 

Further, recent developments in several South Asian states have 
thrust the region's problems more prominantly into public view. 

Among those recent developments is the pro-Soviet coup 
in Afghanistan coupled with signs of increased Soviet influence 
within the new regime and subsequent internal turmoil. Another 
situation causing increased concern was the overthrow of the 
Shah of Iran and resulting uncertainty over the direction in 
which the new government would evolve. Adding to general con- 
cern for the course of the Iranian situation has been the 
centrifugal forces set in motion by Kurdish and Arab national- 
ists as well as politically liberal forces within the country. 
In addition, the execution of Bhutto by Pakistani authorities 
has increased internal dissent in that country. This dissent 
is significant because it occurs during a period of tension 
between Afghanistan and Pakistan concerning reputed Pakistani 
support for dissident tribesmen challenging the Taraki regime 
in Afghanistan . 3 "*" 



30 

A regional perspective on area tensions is presented by 
Amire [Ref. 66], and Misra [Ref. 67]. Burrell and Cottrell 
[Ref. 68] explore specific problems faced by the states of 
South Asia. 

31 These developments are being continuously reported in 
magazine and newspaper articles. Articles describing these 
specific developments include : On the Soviet involvement and 

turmoil in Afghanistan; Sherwell, Chris, "Fighting May Boost 
Soviet Presence in Afghanistan," The Christian Science Monitor , 
p. 6, Dec. 29, 1978, and Sherwell, Chris, "Soviet Union Mired 
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For these reasons it was decided to develop a problem 
scenario involving relations in South Asia. The study group 
would be familiar with the basic situation existing in the area, 
and could easily identify important (as well as plausible) 
developments reflecting U.S. interests and possible response 
options. The actual scenario was developed as follows, 
a. Background 

This section, supported with newspaper clippings, 
presented the current situation existing within and among the 
three regional actors being considered. Afghanistan was 
described as having recently undergone a pro-Soviet coup. Its 
military was identified as a prime factor in maintaining poli- 
tical control. The country's administrative bodies, including 
the military, were reported to be heavily influenced by Soviet 
advisors. In addition, the state’s internal situation was 
characterized as being plagued with anti-government groups, 
consisting mainly of Moslem tribesmen in the rural areas. 

The situation in Pakistan was described as 
involving increased internal dissent resulting from Bhutto's 
execution. Problems were also reported to be developing 
from the influx of Afghan refugees. In addition, the 
Pakistani military was described as poorly equipped and 
preoccupied with the Indian threat to Kashmir. 

in Afghanistan," The Christian Science Monitor, p. 3, May 24, 1979. 
On Iran's internal power struggle; Godell , Geoffrey, "Leftists 
Step up Struggle for Control of Iran Revolution," The Christian 
Science Monitor , p. 3, Feb. 16, 1979, and Allaway, Tony, "Kurdish 
Unrest Tests New Iran," The Christian Science Monitor , p. 3, 

Mar. 22, 1979. On Pakistan; "Delayed Decision on Bhutto to 
Affect Pakistan Deeply," The Christian Science Monitor , p. 4, 

Mar. 19, 1979, and Marshall, Tyler, "Indian Subcontinent Faces 
Pressure for Arms Buildup," The Los Angeles Times , p. 1, 

April 22, 1979. 
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•^ ran s current government was described as 
disorganized, composed of several rival factions, and 
challenged by elements (including leftists and ethnic 
minorities) desiring more control. The effectiveness and 
loyalty of Iran s military was also brought into question, 
due to its history of supporting the Shah and the execution 
of several important leaders. This development stressed the 
fact that authority in Iran could still be considered to 
be in a state of flux. Finally, Iran's historic role in 
controlling disputes between Afghanistan and Pakistan 
concerning the territory of Baluchistan was mentioned. 

Using this simplified background discussion as a basis, 
a hypothetical sequence of developments was proposed to 
the study group . 

b. Event Sequence 

- Tribesmen in Baluchistan (South Afghanistan) 
begin a national autonomy movement. This movement 
receives some support from related tribes in Iran and 
Pakistan. 



- Clashes occur between Afghani government 
forces and rebel tribesmen in Baluchistan. 

- The Afghani government dispatches additional 
troops to control the area. These troops are Soviet 
supported, but the extent of this support is unknown. 

- These forces drive the majority of the rebels 
across the border into Iran and Pakistan. 

- Both sides (rebels and Afghani troops) begin 
cross-border raids. By this time the majority of the 
Afghan army is committed. 

- Afghanistan charges that Iran and Pakistan 
are aiding the rebel forces. The Afghan government vows to 
defeat any outside threat to its territorial integrity. 
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The Soviet Union supports this delcaration and offers to 
increase assistance to Afghanistan. 



- Iran authorities deny the Afghani charges and 
state that increased Soviet involvement would only worsen 
the situation. Pakistan charges the Soviet Union with trying 
to precipitate a crisis in the region, and requests U.S. 
aid. 



Regular Afghani forces clash with Iranian 
k° r( 3er troops. This clash erupts into uncontrolled shelling 
along the border. 

- Baluchistan rebels call on Muslim states for 
support against the communists, suggesting a "holy war" 
be launched. 



- Iranian leftists, reportedly supplied by the 
Soviets, begin a terrorist campaign to oust the Ayatollah's 
government . 



- Iran announces support for rebel tribesmen 
and launches a "limited" invasion to pacify the area, and 
expel "expansionist" communists. 

- Pakistani forces join the conflict. 

- The Soviet Union sends troops to Afghanistan 
which check the Iranians. Reports indicate the Soviets 
have -mobilized on Iran's border, that their agents are 
agitating Iranian Kurds, and that they have sent a secret 
mission to Iraq. 

- Soviet and Afghani forces press attack to the 
original Afghan border, but show no signs of halting. 

Steps are being taken to set up a "puppet" government for 
Baluchistan. Indications are that Soviets and Afghanis 
may attempt to include Baluchistan land in Iran and 
Pakistan under control of an "autonomous" Baluchi government. 

Based on these developments, the following 
situation was proposed to exist in South Asia. This situa- 
tion described the crisis problem to be evaluated using 
cross-event analysis. 

c. Resulting Situation 

- Open warfare exists in the area south of 
Afghanistan extending into Iran and Pakistan. This 
fighting involves Afghanistan, Iran, Pakistan, rebel 
tribesmen, and (reportedly) some Soviet forces. 
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The Soviets have mobilized along Iran's 
border and are conducting secret talks with Iraq. 

“ Iranian Kurds and leftists (separately) 
almost in open revolt. 

- India has mobilized along Pakistan's border, 
to "prevent the conflict from spilling over." 

- Chinese troops have mobilized in Sinkiang 
in response to "Soviet aggression, and to demonstrate 
support for the Baluchistan People's Movement". 

To guide the respondents in proposing candidate 
events the following "probable" U.S. concerns were also 
presented . 



d. U.S. Concerns 

- The current conflict threatens navigation in 
the Straits of Hormuz. In addition, the fighting is close 
to the main oil producing region in Iran, threatening the 
security of those fields. 

- The Soviet Union may extend its influence to 
the Indian Ocean if it manages to sponsor a client state of 
Baluchistan. This would permit both overland access to 
the Indian Ocean and possible naval basing rights. 

- The current turmoil directly threatens U.S. 
civilians in the area and, to a lesser extent, U.S. economic 
interests . 



- The potential involvement of India and China 
and possible direct Soviet intervention suggest a major 
confrontation may be developing. 

This scenario provided the background against 
which the respondents were asked to suggest a candidate 
event set, and formulate the required subjective estimates. 
This process will be described in the following section 
which details the implementation design's third phase as 
it actually developed. 



2. Elicit Estimates 



The next task undertaken in the demonstration re- 
quired establishing a candidate event set. Before the 
9 rou P was asked to brainstorm possible developments, several 
procedural points were reviewed. The study group was 
reminded that suggested events be considered as developing 
from the given situation. Enzer's criteria for including 
events in the set were also reviewed. These criteria 
stress that candidate events be (1) important to the situa- 
tion, (2) of uncertain outcome, (3) causally related, and 
(4) specific. Finally, the event formulation rules proposed 
by Mitchell and Tydeman were reviewed to assure the suggested 
events could be clearly understood by the group. These 
procedural points were discussed in Chapter Two. 
a. Establishing the Event Set 

Once the group understood the required event 
formulation method, a brainstorming session began. As 
events were suggested they were recorded and posted before 
the group. This process continued until it became apparent 
that the participants had exhausted all their ideas. A 
brief discussion session then was allowed to clear up any 
uncertainties. Twenty-nine candidate events were proposed 
during this session. Ten of these represented possible U.S. 
responses to the situation, the remainder represented situa- 
tion developments. These events are presented in Appendix 
A. 



In order to select an analysis event set the 
group was next asked to provide the subjective evaluations 
required for Mitchell and Tyderaan ' s "event assessment" 
procedure. These inputs consisted of two estimates. The 
first required the respondents to evaluate each event's 
initial occurrence probability. These probabilities were 
collected individually by having each respondent score every 
event on the following scale. Each evaluation was to be 
made without considering the influence any other event in 
the set might have on the situation. 



TABLE TWO 



EVENT PROBABILITY DESCRIPTIONS AND RATINGS 



STATEMENT 


RATING 


Won ’ t Occur 


. Q5 


Very Unlikely 


.15 


Unlikely 


.30 


Fifty-Fifty 


.50 


Likely 


.70 


Very Likely 


.85 


Certain 


. 95 



The second estimate required respondents to 
evaluate each event's significance should it occur. An 
event's significance was defined as the influence it would 
have on the overall problem situation, assuming it did 
occur. These ratings were based on the following scale, 
and were again collected individually. 
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TABLE THREE 



EVENT SIGNIFICANCE DESCRIPTIONS AND RATINGS 



STATEMENT RATING 

No Importance 1 
Some Importance 2 
Important 3 
Very Important 4 
Critical 5 



This procedure produced ten values under each 
criterion for each of the twenty-nine events . These indi- 
vidual scores were aggregated and used to produce an impact 
rating for each event. The twenty-nine candidate events 
and their probability, significance, and impact ratings are 
contained in Appendix A. 

Several problems were encountered during this 
estimation procedure which were unanticipated in the prob- 
lem design. While eliciting candidate events two tenden- 
cies were noticed which seemed to inhibit the formulation 
process. In brainstorming, the respondents seemed to con- 
centrate on a suggested event and propose subsequent events 
which would develop should the first event occur. Often 
this process would lead far afield until interrupted by 
the facilitator interjecting an alternative event. On 
other occasions, an event's suggestion seemed to elicit the 
proposal of a response event aimed specifically at the first 
event. Another problem encountered stemmed from the respon- 
dent's experience level or knowledge. 
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Some respondents seemed to be intimidated by 
the ease with which other respondents formulated and jus- 
tified suggested events. A few respondents stated they 
felt unqualified to participate, and questioned the validity 
of their future estimates given their lack of equal knowl- 
edge concerning South Asia. These problems seem to be 
inherent in the "brainstorming" process. 

In future applications it is felt that using 
trained or experienced facilitators could reduce the prob- 
lem of respondent intimidation, and better control the brain- 
storming process. In addition, research using the delphi 
procedure indicates that having respondents rate themselves 
on their confidence in answering questions provides a cri- 
terion with which to weight individual responses to produce 
improved group values [Ref. 13] . This method might be 
incorporated in future cross-impact applications. 

The fact that suggested events often led to 
related events being suggested indicates a basic cross- 
event assumption may be unfounded. The approach operates 
on the theory that respondents can think in terms of dis- 
crete events. Based on this assumption respondents are 
asked to provide initial probability and significance esti- 
mates for each event independently. However, during the 
brainstorming process, respondents exhibited a tendency to 
propose events in an iterative manner. This suggests that 
the group may find it difficult to conceptualize relationships 
in terms of discrete events, and therefore may conceive of 
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of related probability and significance questions inter- 
dependent^, or as contingencies. This in turn indicates 
that another approach to quantifying estimates may be more 
appropriate. This point will be developed more exten- 
sively in the next chapter. 

Other problems were encountered when the indi- 
vidual estimates were aggregates to produce the impact 
ratings. In their procedure, Mitchell and Tydeman use a 
weighted product of each event's probability and signifi- 
cance scores to produce its impact rating. This procedure 
was found to produce inadequate results in this demonstra- 
tion. The results obtained seemed to disguise the impact 
assigned an event which received a combination of a low 
and high rating on the two criteria evaluated. A simpli— 

hypothetical example may help to illustrate this point. 

Assume three countries are rated on their mili- 
tary potential by aggregating scores for their Army, Navy, 
and Air Force. Using a multiplicative index produces the 
following results. 



FIGURE 4 

RESULTS OF A MULTIPLICATIVE INDEX 



MILITARY 

COUNTRY ARMY (x) NAVY (x) AIR FORCE ( = ) POTENTIAL 



A 


5 


2 


3 


30 


B 


5 


1 


3 


15 


C 


5 


0 


3 


0 
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Using the same values with an additive index seems to 
produce a more realistic measure. 



FIGURE 5 

RESULTS OF AN ADDITIVE INDEX 



MILITARY 

COUNTRY ARMY (+) NAVY (+) AIR FORCE (=) POTENTIAL 



A 


5 


2 


3 


10 


B 


5 


1 


3 


9 


C 


5 


0 


5 


10 



Another, less significant, problem developed 
because two different scales were employed in the "event 
assessment." Probability values were quantified between 
zero and one, while significance scores ranged from one 
to five. This further distorted the multiplicative index. 
As a result the impact scores produced an analysis event 
set which did not seem to accurately reflect the event s 
relative importance. To overcome this problem two correc- 
tions were introduced to the aggregation procedure. 

The significance ratings were rescaled to 
fall between zero and one. The final ratings employed 
were: 5=.95, 4= . 7 , 3=.5, 2 — . 3 , and 1=.05. Then the 

individual probability and significance ratings were 
summed to produce an individual event impact rating. 

All three ratings were then averaged to produce a group or 
mean score, and the standard deviations were computed -or 



each measure . These are the values presented in Appendix 

A. 

This correction is not without conceptual 
limitations . The assumption implicit in summing the two 
evaluation measures is that both probability and signifi- 
cance are equally important in determining an event's 
impact on the problem environment. It is doubtful whether 
this assumption would hold up under rigorous examination. 

It appears that an event's significance would be a more 
important consideration than its occurrence probability. 

This is because an event's impact has no meaning should it 
not occur, therefore, it only becomes important from a 
policy standpoint once it occurs, and this importance 
derives from its significance to the situation. This prob- 
lem relates to Gordon's discussion of the complex factors 
which make up an event's cross-impact [Ref. 24]. Presently 
there exists no algorithm relating the many factors (such 
as probabilities of "feasibility" and "pursuit", and event 
significance) which compose an event's impact. As a result 
simple summing was the only approach which could be readily 
applied . 

Candidate events were ranked by impact scores 
and six events selected to form the analysis set. A final 
ranking by event impact scores is contained in Appendix A. 
Because one purpose of the demonstration was to examine the 
cross-impact technique's applicability to evaluating specific 
policy response options these six events were not chosen from 
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the top ranked candidate events . The six top ranked events 

included only one event identified as a response option 

(number three) . This was because, while most response 

events were evaluated as having high initial probabilities, 

3 2 

their significance was evaluated as being relatively low. 

To obtain a matrix containing an equal number of response 
events and situation development events, the events were 
separated and the top three chosen from each category. 

The analysis event set and the mean initial 
probabilities are presented below. Six events were used 
because this was felt to be the maximum number which could 
be effectively manipulated by hand. It should be noted that 
the initial probability values have been rounded to two 
decimal places. The averaging technique used retained three 
decimal place values for comparison purposes. Because 
these values were averaged subjective estimates, this 
degree of accuracy would not be jsutified for use in the 
actual demonstration. This event set formed the basis for 
the remaining individual estimates , that is , the five 
causal time-dependent relationships developed by Mitchell 
and Tydeman . 



32 This result is interesting due to its implications. 
Although the U.S. is assumed to have an extensive number 
of responses available which could be employed in a given 
situation, the respondents obviously felt that the majority 
of these would be ineffective in this case. Of the three 
response options included in the analysis, only one had a 
significance score comparable to that of the situation 
development events. 



TABLE FOUR 



SELECTED EVENTS 



No. 


STATEMENT 


Avg . P ^ 


22 


Soviet/Afghan forces gain control of 
Baluchistan . 


.60 


28 


Soviet Union builds up Indian Ocean 
squadron. 


• 

CX) 

o 


23 


Soviet Union airlifts supplies to 
Afghanistan . 


.72 


3 


United States deploys CTF to 
Indian Ocean. 


.76 


9 


United States evacuates civilians 
from area. 


.84 


8 


United States offers to negotiate 
a cease-fire. 


.64 



b. Estimate Relationships 

The six-event set produced fifteen possible 
event pairs which required further estimation. The 
required estimates were the five time-dependent probabili- 
ties (C 1 through C 5 ) which assessed occurrence, non- 
occurrence, and joint occurrence. Thus, seventy-five 
estimates were required from each respondent. 

In order to systematically produce and collect 
these estimates , response forms were duplicated from the 
examples provided by Mitchell and Tydeman [Ref. 29, p. 146— 
147]. The forms used are reproduced on the next page. 
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FIGURE 6 



FORM FOR EXTIMATING C-^ AND 



Would the prior occurrence of event # cause your estimate 
of the Probability of event # to: 



Increase 



Stay the same 



Decrease 



If your estimate of event # probability is changed (increased 
or decreased) is the magnitude of the change: 



Only Slight 



A Little A Fair Bit 



A Lot Sufficient to 
make event # 
certain or 
impossible 



5 10 

(% of total possible change) 



25 



50 100 



FIGURE 7 

FORM FOR ESTIMATING C n AND C. 

2 4 

Would the prior non-occurrence of event # cause your 
estimate of the Probability of event # to: 

Increase Stay the Same Decrease 



If your estimate of event # probability is changed ([increased 
or decreased) is the magnitude of the change: 

Only Slight A Little A Fair Bit A Lot Sufficient to make 

event # certain 
or impossible 

5 10 25 50 100 

(% of total possible change) 
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FIGURE 8 



FORM FOR ESTIMATING C r 

5 



Assuming both events # and # occur within the next 14 days 

Event is certain to occur first 
Event is very likely to occur first 
Event is likely to occur first 

Event and event are equally likely to occur first 
Event is likely to occur first 
Event is very likely to occur first 
Event is certain to occur first 



These forms were organized by selecting the 
first event in the set and pairing it with the remaining 
five events. The next series of estimates paired the second 
event with the remaining four events and so on until all 
fifteen event pairs had been specified. Because there were 
seventy-five total estimates, the forms were delivered to 
the respondents in two batches. Each respondent was given 
two days to complete each batch. The forms were then 
collected and sorted by event pair and the relationships 
assessed. The ten responses for each relationship were 
averaged to produce a mean score for each relationship. 

These scores are contained in Appendix A. 

Several complaints were voiced by the respon- 
dents concerning this estimation procedure. This approach 
made the estimation process monotonous. Three questions 
were repeated with only different event numbers 
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being inserted to complete the seventy-five data points. 

In addition, the question wording was felt to be tortuous, 
specifically through use of the terms "prior occurrence", 
and "prior non-occurrence". 

These problems result from manually replicating 
the cross-event procedure. It was not possible in the 
time available to devise an optimal question format for 
each event pair. Such a format could have incorporated 
the event statements within the question. For example 
the following formulation for event pair ( 28 , 3 ) would 
probably have been more acceptable: 

"Assuming the Soviet Union builds up its 
Indian Ocean fleet, would you estimate 
the probability that the U.S. deploys 
a CTF to the Indian Ocean to:" 

and for non-occurrence: 

"Assuming the Soviet Union does not 
build up its Indian Ocean fleet, would 
you estimate the probability ..." 

Such a format could obviously be constructed for each 

event pair. Several advantages might stem from adopting 

this approach. The more precise wording explicitly 

incorporates the event statements being considered. The 

causal time-dependent nature of the relationship being 

explored is implicit in the question without employing 

phrases such as "prior occurrence." As a result the responses 

obtained might be more accurate and consistent, however, 

this fact cannot be validated without further analysis of 

how respondents interpret questions. 
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Another problem encountered in the procedure 
was not voiced by the respondents, but was hypothesized 
to account for the responses obtained when the group values 
were analyzed. The analysis of the responses presented in 
the next section found that every event pair was inconsistent 
in terms of the through Pg ordered outcome probabilities 
computed. This problem was felt to result, at least par- 
tially, from the sequence in which estimates had been 
obtained . 

The sequence used in this demonstration was 
selected because it systematically specified all the event 
pairs. This was necessary to insure that all respondents 
received the proper forms and thus provided all required 
estimates. However, this may have been an inappropriate 
way to procede through the event pair set for estimation 
purposes. In addition, the five relationships being esti- 
mated were not presented to the respondents in order. The 
joint occurrence question (C^) being retained until the 
end of each series (Event 1 | Event 2,..., event 6) of the 
through questions. 

As the respondents answered the questions in 
the order of presentation, it is possible that they lost 
track of the values previously specified. It is also 
possible that the respondents were prevented from con- 
sidering their previous responses concerning event pairs 
which might have been felt to be related to each other . 

Once again these possibilities cannot be verified without 
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experimental analysis. No doubt, there is an optimal way 
to present the event set for estimation. 



This conclusion is based on two arguments pre- 
sented in the cross-impact theory discussion. First, Enzer 
suggested that probabilities of conditional relationships 
were bounded by the probabilities previously specified for 
that event pair. If this is true then this condition could 
also apply to the time-dependent causal probabilities. The 
second argument is even more basic to cross-impact theory. 

Gordon suggested that most events are related 
in some way. Enzer approached this conclusion in his dis- 
cussion of coupling. It therefore seems reasonable that the 
discrete events composing a set may all be coupled in some 
way. This, it should be noted, is not the concept of higher 
order interactions which Turoff attempted to measure, but 
refers to direct relationships among all events in the set. 
Assuming such a situation is present, then in order to pro- 
duce a consistent set of estimates, a respondent needs to 
be able to review and incorporate not only his previous 
estimates within an event pair but among event pairs as well. 

In summary, the problems encountered and sus- 
pected in this process were: Cl) monotony, (2) imprecise 

question wording, and (3) inappropriate question sequence. 

To overcome these problems it would be necessary to devise 
an estimation procedure which: (1) formulates questions using 

the event statements, (2) develops a more exact question 
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structure, (3) allows estimators to review their responses 
as they estimate, and (4) allows estimators to compare 
their responses across related event pairs. 

One possible approach to overcoming these prob- 
lems might be to use an interactive computer program to 
elicit estimator responses. This approach was demonstrated 
by Turoff. As proposed here, the program would allow the 
respondent to make estimates on the basis of responses con- 
cerning other estimates. Obvious extensions to this approach 
include computing probability boundaries as suggested by 
Dalkey , and displaying results to allow re-estimation as 
suggested by Turoff. Using an interactive computer would 
aid in obtaining consistent probability estimates (this 
problem will be returned to in a later section) . It might 
also provide useful data for determining how respondents 
interpret questions posed, and for devising an optimal 
estimation sequence. 

Despite the problems encountered by the respon- 
dents, and because the suspected problems were not immediately 
apparent, the respondents' estimates were collected and 
averaged as planned. These values formed the input for 
computing the eight ordered outcome probabilities for each 
event pair. 

3 . Analysis 

The next phase in the demonstration was to replicate 
the analyses performed by Mitchell and Tydeman as extended 
for this application. This required that the responses 
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obtained be converted into conditional probabilities, and 
"that these be used to construct a cross— event matrix and 
generate scenarios . This section will describe the compu- 
tational procedure used and discuss the results obtained, 
a. Computational Procedure 

Once the respondents' estimates were averaged 
and the standard deviations for each relationship computed, 
mean values were used to compute the ordered outcome proba- 
bilities. The through Pg values for each event pair 
were calculated using the formula derived by Mitchell and 
Tydeman as presented in Section E of Chapter Three. These 
values were used to compute the conditional probabilities, 
however, unlike the Mitchell and Tydeman demonstration, 
every event pair was found to have inconsistent results . 

According to Mitchell and Tydemand inconsistency 
exists when any outcome probability is negative [Ref. 29, 
p. 142]. In fact pairwise inconsistency from a probabilistic 
standpoint is manifested by a value >1 or <0, and is indi- 
cated when the common factors a, 3, and y exceed unity or 
become negative. The calculated outcome probabilities for 
each event pair are presented in Appendix B. 

The cross-event procedure uses a minimum stan- 
dard deviation fitting technique to resolve pairwise incon- 
sistency. This technique, as described in Chapter Three, 
was felt to be inappropriate for use in this demonstration. 
One reason was that it would require an excessive amount of 
time to resolve fifteen inconsistent sets. Also, the 
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technique, as demonstrated by the developers in a case 
resolving individual inconsistency, zeroed the inconsistent 
values. Of the fifteen event pairs developed in this study, 
over half the outcomes in twelve pairs were inconsistent 
while six pairs had seven or more inconsistent values. As 
a result, it was felt that the fitting technique would 
destroy too much potentially useful information. 

Even though the ordered outcome probabilities 
were apparently inconsistent, it was still possible to com- 
pute conditional probabilities using the formulae devised 
by Mitchell and Tydeman. These conditional probabilities 
would doubtlessly also be inconsistent from a probabilistic 
standpoint, however, in the cases where one conditional 
probability appeared consistent it would be possible to 
employ Enzer ' s boundary rule to resolve the second proba- 
bility. As in the case of the fitting technique, this pro- 
cedure was expected to destroy at least half of the potential 
information. In addition, the boundary rule could not be 
applied in cases where both conditional probabilities appeared 
to be inconsistent. 

Because the demonstration results were apparently 
inconsistent, it was not possible to produce all the data 
required for the intended analysis and evaluation. Having 
only the ordered outcome probabilities precluded any attempt 
to determine event dominance and sensitivity. Nor was the 
required data produced for completing the cross-event matrix 
and undertaking scenario generation. Therefore, it was not 
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possible to conduct the analyses which use these tools. In 
P art icular , the demonstration could not be used to determine 
the validity of the proposed extension which included dis- 
crete policy options in the event matrix. Even so, it is 
j-slt that this was a practicable extension, given a 
workable cross-impact format, which could become a useful 
analytical tool. 

Two alternatives for solving the inconcsistency 
problem seemed to present themselves. The designed study 
could be continued by arbitrarily specifying a consistent 
set of values. This alternative would have ignored the 
study's intended purpose of assessing the utility and 
effectiveness of cross-impact analysis as a possible 
decision-aid by forcing the methodology to work rather than 
examining its problems and their implications. 

The second alternative was to examine the appar- 
ent inconsistencies to determine if they were in fact incon- 
sistent. It would then be possible to determine; a) why 
the information was inconsistent and what corrective measures 
could be taken, or b) why the information appeared to be 
inconsistent, and how it could be used if it was in fact 
consistent. This alternative was selected. The approach 
used will be described in the following section. 

b. Exploring the Inconsistency Dilemma 

The first task in exploring the inconsistency 
problem was to determine if the demonstration results were 
in fact inconsistent. To do this the cross-event approach's 
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assumptions and implications were re-examined. It was 

obvious that the collected data provided enough information 

to rank the ordered outcome probabilities. If the outcome 

Probabilities were considered as an eight element set there 
8 _ n! 

were P g - , = 8! possible rank combinations. If it 

could be shown that the fifteen combinations obtained during 
the demonstration formed a pattern displaying some underlying 
logical consistency apart from the consistency demanded by 
the constraints of probability theory, and if this pattern 
occurred in a statistically significant number of cases, 
then the responses could be demonstrated to provide useful 
information which might otherwise be masked by routine 
cross-impact analysis techniques. 

The respondents had been asked for causal, 
time-dependent estimates which were used to compute ordered 
outcome probabilities. The estimates were obtained by 
assuming the certain occurrence or non-occurrence of an 
event (cause) and assessing the change in probability this 
produced in another event's probability (effect). The out- 
come probabilities computed using these estimates were time 
sequences, which retained this cause and effect structure. 

The eight ordered outcomes could, therefore, be paired by 

33 

opposite cause. This resulted in pairing the following 



Actually, several pairing methods were possible. For 
example: Transpose (i,j with j,i)_j_ opposite cause ( i , j with 

i,j), opposite effect (i,j with i,j), and opposite cause and 
effect (i, j with i,j). However, since the relationship being 
estimated assumed known cause (prior occurrence or non- 
occurrence), in effect holding cause constant, the opposite 
cause pairing was considered more relevant for testing. 
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outcomes: P x - P 3 (i,j with i,j), P 2 - p 4 (i,J with 1,1), 

P 5 “ P 7 ( ^ ,:L Wlth 3ri), and P g - p g ( j , T with 7,1) . if 
the respondents had been consistent, it could be argued 
that the pairing of probability ranks by cause would reveal 
some logical pattern. To accept this argument, the pattern 
would have to appear more frequently than could be expected 
by random chance. 

Upon pairing the probability rankings in the 
fifteen cases (one for each event pair) by cause, several 
patterns were noted. These are depicted in Appendix B. 

As stated above 8! combinations are possible. To show con- 
sistency a pattern would have to permit prediction (i.e., 
by knowning one outcome probability's rank it would be 
possible to determine its pair's rank.) By this criterion 
one obviously consistent pattern was noted and termed 
the "Chinese Box". 

The "Chinese Box" pattern paired the first and 
last, second and seventh, third and sixth and, fourth and 
fifth ranked probabilities. The pattern was intuitively 
pleasing as it indicated that the respondents had ranked 
last the opposite cause to the cause ranked first, ranked 
next to last the opposite cause to the cause ranked second, 
and so forth. This pattern implied that some logical con- 
sistency existed in the way respondents linked cause and 
effects in the demonstration situation. A second consistent 
pattern could also be identified although it was intuitively 
less rigorous. This was actually a family of patterns which 
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existed when one pair and its opposite pair were located 
in the upper half and lower half of the scale respectively. 
This also implies some degree of consistency although it 
does not allow exact prediction. The "Chinese Box" pattern 
was selected for testing because it was more rigorous, and 
could be expected to occur less frequently by random chance 
than the simple upper-lower split. 

A "Chinese Box" occurs when one of the outcome 
is ranked in position one through four before its 
pair is ranked in the corresponding position from the bottom 
up (eight to five). Thus there are = ij- = 4! combinations 
which produce this pattern. But since can be replaced 
by P 3 / P 2 can be re P lacad - by P 4 , P 5 can be replaced by P , 
and ^6 can be re Pb ace d by Pg, the actual number of combina- 
tions is given by 4! x2x2x2x2= 384. Thus the 
theoretical probability of observing a "Chinese Box" can 
be determined as 384/81 = .0095 = .01. During the demon- 
stration the following results were obtained. In the fif- 
teen cases, eight obviously inconsistent sets occurred and 
seven apparently consistent sets were observed. This pro- 
duces an experimental frequency of .47. Two tests were 
used to determine if this result was statistically signifi- 
cant. 

The Kolmogorov-Smirnov one sample test provided 
the following results for a sample set of N = 15 cases, 
and two categories (consistent, inconsistent) [Siegel, 1956, 
p. 47-52]. 
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F o (X) the theoretical cumulative distribution 

of cases 

^ 15 = the observed cumulative frequency dis- 
tribution of a random sample of 15 cases 

D = the maximum deviation between F (X) and 
S 15 (X) or |F q (X) - S 15 (X) | ° 

The critical value for D at the .01 significance level is 
.404 for 15 cases [Siegel, 1956, Table E, p. 251]. 



TABLE FIVE 

CONSISTENCY PATTERNS IN 15 CASES 



f = number of 
cases in 
each category 


OUTCOME EVALUATED AS 


CONSISTENT 


INCONSISTENT 


7 


8 


V x) 


.15/15 


15/15 


S 15 (X) 


7/15 


15/15 


D = |F q (X) - S 15 (X) | 


6.85/15 = .456 


0 



Since D = .456 > .404, this finding is evaluated as signifi- 
cant at the .01 level. However, as in most cases involving 
a small N this conclusion would not be justified if the 
number of consistent outcomes was reduced by one. For six 
consistent outcomes D = .39 and is only significant at the 
.05 level where the critical D is .338. 

As an additional check on the statistical signi- 
ficance of this outcome and in order to determine the probability 
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that this demonstration represents a random sample, the 
Binomial Test was applied [Siegel, 1956, p. 36-42]. In 
this test: 

P = proportion of cases expected to be consistent 

Q = 1 - P = proportion of cases expected to be 

inconsistent 

N = number of cases 

x = number of observed consistent outcomes 

The probability of observing exactly seven consistent out- 
comes is given by: 

N x N-x 15 7 8 

(£)P Q = p(7) = ( 7 ) (.01) (.99) 

.0000000001 



P (x) = 



where 



Thus p (7) =0, and the probability of observing five or more 
consistent outcomes can be computed similarly as p (x >_ 5) 

= .0000004. Since these tests both indicate that the observed 
frequency of consistent rankings (.47) was significant, 
the following conclusion was reached. 

Although all event pairs were evaluated as being 
inconsistent using the cross-event approach's probabilistic 
criteria (any through Pg < 0 or > 1) , in actuality seven 
cases were consistent when the probabilities were ranked 
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and paired by opposite cause. The implications of this 
finding will be addressed in the concluding chapter. 



D. CONCLUSIONS 

This chapter described the implementation process de- 
signed to demonstrate cross-event analysis in a hypothetical 
crisis situation. Several problems were encountered in 
Pitting the design into effect. During the estimation 
phase it was discovered that a "brainstorming" session 
could get out of control and range far from the intended 
situation, or intimidate some respondents. It also seemed 
that the respondents had difficulty thinking in terms of 
discrete events. Furthermore, the method suggested for con- 
ducting "event assessment" was found to be inappropriate 
because it used two different scales and a multiplicative 
index. 

Other problems were encountered in estimating the 
interrelationships between event pairs. Some problems arose 
due to limitations in the implementation design, specifically, 
the problems of monotony and imprecise question wording. 
Because inconsistent probabilities were obtained, addi- 
tional problems were suspected to exist in the estimation 
process. These problems involved the sequence in which the 
estimates were asked for and the fact that respondents might 
have lost track of their previous responses in the course 
of completing the forms. Finally, the way in which rela- 
tionships were quantified was suspect. 
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To overcome these problems, several solutions were 
proposed. An experienced facilitator was recommended to 
control the brainstorming process. In addition, incorporating 
a method tor respondent self-evaluation was suggested to 
permit weighting estimates for reliability. Because respon— 
dsnts apparently had difficulty conceptualizing discrete 
events it was suggested that they thought in terms of inter- 
relations or contingencies. 

Improvements to the "event assessment" process included, 
using similar scales for the measures being combined, and 
adopting an additive impact index rather than a multiplici- 
tive one. It was also noted that further research into the 
components which constitute an event's impact, and their 
relative importance was also desirable in order to devise 
an appropriate algorithm for combining the various measures. 
Overcoming the problems of monotony and imprecise question 
wording in assessing interactions requires devising an 
improved question format. 

This improvement could be realized utilizing an inter- 
active computer program. This would also allow respondents 
to review their estimates, and could be extended to provide 
probability bounds for estimates, and to present respondents 
with a display of the results obtained from their inputs. 

Such programs have been demonstrated in several applications 
reviewed in this study. The other problems suspected in the 
estimation process were only hypothesized and could not be 



157 



verified without additional study. Data regarding question 
sequence and respondents' thought processes could be col- 
lected while using the interactive computer program. 

Once the implementation problems were discussed and 
possible solutions suggested, the experiment results were 
presented and analyzed. It was found that the ordered out- 
come probabilities for each event pair, as computed from 
the respondents' estimates using the rules of probability 
theory, contained inconsistent values. However, when the 
information required from the respondents was presented as 
causal, time-dependent estimates, and the information con- 
tained by the computed output was presented as the respon- 
dents' view of the sequence in which the two paired 
events were likely to occur, another interpretation was 
possible . 

The ordered outcome values could be ranked and paired 
according to opposite cause. In seven of the fifteen 
cases, this procedure produced a pattern suitable for exact 
prediction. This "Chinese Box" pattern was shown to be 
intuitively pleasing as an indication that the respondents 
had been logically consistent in their responses . 

In order to determine if this result was statistically 
significant, two tests were applied to the data. The 
Binomial Test showed that the probability of obtaining more 
than five "Chinexe Boxes" was close to zero, while the 
Kolmogorov-Smirnov one-sample test showed that the results 
obtained were significant at the .01 level. Therefore, it 
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was decided that the respondents had been consistent in 
almost half of the event pairs, and that the apparent 
inconsistency in those cases resulted only because the 
values computed did not produce valid probabilities. 

The next chapter will explore the implications of this 
finding for the future use of cross-impact analysis. This 
will require evaluating the approach's basic assumptions 
regarding how interactions are quantified and manipulated. 

This evaluation will draw on the impression formed during 
the brainstorming process that respondents may think in 
terms of interrelations or contingencies. The future utility 
of cross-impact analysis as a crisic decision-aid will be 
assessed using the experience gained in applying the tech- 
nique in this demonstration, and through drawing on the 
feedback comments made by the study participants. Finally, 
it is hoped to suggest possible improvements to the technique, 
and to recommend further areas for study. 
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V. CONCLUSION 



A. INTRODUCTION 

Aside from the several minor problems encountered in 
implementing the selected cross-impact analysis technique 
in a hypothetical situation, this study revealed a major 
conceptual limitation in the technique . This conceptual 
problem underlies a basic assumption which must be adopted 
in any cross-impact approach. The problem stems from 
determining what interpretation should be applied to the 
estimates provided by the expert respondents. This problem 
relates to determining how individual thought processes 
actually work, and how complex relationships are concep- 
tualized. The problem underlies any methodology which 
attempts to employ subjective estimates. 

In reviewing cross-impact analysis' basic concepts 
and assumptions it was noted that several interpretations 
could be applied to subjective estimates, depending on the 
assumption made concerning the way individual's interpret 
and analyze unfolding situations. These alternative inter- 
pretations can be grouped into at least two main categories. 
The first, which was pursued in this study, viewed the 
estimates as causal probabilities. Under this interpreta- 
tion, individuals are assumed to be essentially Bayesian 
information processors. 
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The second interpretation views subjective estimates 
as quantified variation estimates, or, as directly describing 
how a change in one variable produces a change in another 
variable. Under this interpretation, an individual is 
assumed to be a more complex information processor than is 
described by the Bayesian model. This interpretation was 
used in the Turoff approach, which represented the results 
obtained from the respondents' estimates as analogous to 
correlation coefficients. 

How the question of interpreting subjective estimates 
is resolved forms the basis for the methodological approach 
used in cross-impact analysis. The assumed interpretation 
determines how inconsistency is identified and corrected. 

It also influences the method used to obtain estimates and 
produce cross-impact factors. For these reasons, determining 
the appropriate assumption to be used in interpreting 
estimates is essential to performing reliable, useful 
cross-impact analysis. 

The results obtained in this study indicate that a 
probabilistic interpretation of estimates may not be appro- 
priate. It was demonstrated that, although the ordered 
outcome values were inconsistent as probabilities, a 
significant number of event pairs contained an outcome 
pattern which indicated that a consistent thought process 
had occurred. Additionally, during event generation, 
respondents showed a tendency to think in terms of contin- 
gencies or interrelations. 
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This chapter will address the question of how respondent 
answer the cross— impact questions posed in light of the 
study results. This discussion will focus on the alter- 
native interpretations proposed, and how each of them may 
be used in future cross-impact analysis applications. The 
final section of this chapter will discuss the technique's 
use as a decision-aiding methodology as it is currently 
structured. This discussion will highlight the technique's 
strengths and weaknesses in problem analysis. 

B. INTERPRETING RESPONDENT ESTIMATES 

Of the two interpretations proposed, the most frequently 
employed is probability. The inherent advantages of this 
interpretation were discussed in the technique selection 
section of Chapter Three. These advantages were primarily 
methodological in nature. The probabilistic assumption 
provided a method for determining if respondents' estimates 
were consistent. Furthermore, by using probabilistic rela- 
tionships, a well structured algorithm for manipulating 
estimates was also developed. 

1 . Probabilities 

Using probability values restricted the estimated 
and computed values to a range between one and zero. 

Several procedures were designed to correct values which 
exceeded these bounds. All of these procedures essentially 
zeroed or made unity the extreme values . As demonstrated 
in this study, use of these correction procedures could 



destroy much information. This is true especially when a 
large number of extreme responses are obtained. This study 
also demonstrated that bounding the estimated values to 
normal probability limits can disguise otherwise consistent 
results. A possible explanation behind the excessive 
number of extreme responses is that the respondents do not 
easily conceptualize events and their interactions in 
probabilistic terms. 

This problem could be reduced if respondents were 
trained to make probability estimates . An interactive 
computer package could aid in this process. As discussed 
previously, such a package could be structured to provide 
respondents with the probability bounds for their future 
estimates based on the values previously specified. The 
program would also furnish the respondents with results 
computed from their estimates. Repeated exposure to such 
outputs could increase the respondents' experience with 
probabilistic relationships. Eventually, they might become 
proficient in making consistent probabilistic estimates. 

However, aside from the previously mentioned 
methodological advantages, there does not appear to be any 
innate justification for pursuing this solution. The results 
of this study indicate that consistent responses can be 
obtained when using other than consistent probability values. 
If respondents do find probabilities difficult to work with, 
yet can produce consistent responses through some other 
thought process, it might be more appropriate to devise an 



analytic procedure which taps expert opinions by using 
some other algorithm. In this case, the second inter- 
pretation given respondents' estimates might be worthy of 
increased attention and analysis. 

2 . Variations 

The second interpretation given respondents' 
estimates suggests that individuals may be more comfortable 
working with a procedure which quantifies the variations 
produced by one variable interacting with another variable. 
The fact that the respondents appeared to think in terms 
of contingencies during event generation lends some support 
to this interpretation. If it can be determined that 
respondents are more comfortable when dealing with relative 
changes in events, a variation algorithm might be used to 
collect and manipulate their estimates. 

Two measures have been developed which describe the 
relationships between variables (in this case events). 

Either, or both might be developed for use in cross-impact 
analysis. Turoff demonstrated the application of one 
variation measure, a correlation-based algorithm, in his 
cross-impact approach. The second variation measure, 
regression, might also be appropriate for use in cross- 
impact analysis. 

Correlation coefficients range between minus and 
plus one. These boundaries could produce constraints similar 
to those encountered in using probabilities. However, using 
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a correlation algorithm would retain a certain ease of 
manipulation. In addition, it would also be possible to 
obtain an indication of estimator consistency. 

This capability derives from the fact that under 
this interpretation the cross-impact matrix would be 
analogous to a correlation matrix. Consistency could 
therefore be determined through analyzing the matrix entries. 
A brief example, using a simplified correlation matrix, will 
demonstrate this point 



FIGURE 9 

SIMPLIFIED CORRELATION MATRIX 
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In a correlation matrix, if variable C increases 
with an increase in variable A; then variable A must increase 
with an increase in variable C. In other words, the matrix 
entries above the diagonal must mirror the entries below 
the diagonal to be consistent. 
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As a final consideration, the theory describing 
correlation coefficients is well enough developed that 
other analysis forms might be attempted. For example, the 
technique of path analysis could be developed as a method 
for tracing causal relationships. It might also be 
possible to trace logical structures on adjacency matrices 
using the principles developed by Harary [Ref. 42], and 
demonstrated by Bonham, et al., and Axelrod [Refs. 44, 45, 
and 43]. The correlation algorithm for cross-impacts might 
also be extended into a regression algorithm. 

Regression coefficients represent the second varia- 
tion measure which might be applied to cross-impact analysis. 
Because regression coefficients can range between plus and 
minus infinity, there would be no constraints imposed by 
bounding estimate values. However, using a regression 
algorithm could easily produce extremely large values, 
limiting the ease with which estimates could be manipulated. 
This fact is apparent upon recalling that some ordered out- 
come values computed in this study exceeded a value of 
thirty, and these were calculated using bounded estimates. 

3 . Summary 

This study's results indicate that the method used 
in cross-event analysis to tap expert's opinions may be 
inappropriate. Two alternatives might be used to overcome 
this problem. The first possibility is to train respondents 
to think in probabilistic terms. The second is to develop 
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another algorithm for interpreting and manipulating their 
estimates, possibly through using correlation or regression 
coefficients . 

Choosing between these two alternatives cannot be 
attempted without further study. The primary focus for 
this additional study would be to empirically determine how 
people actually conceptualize relationships between events. 
Without an understanding of this crucial point, selection 
of a cross-impact algorithm can only be made arbitrarily. 

As a result, arbitrary estimate interpretation would 
preclude giving any confidence to specific results obtained 
from the analysis. Despite this problem, it is felt that 
the methodology has some potential as an alternative 
approach to decision analysis in socio-political 
applications . 

C. CROSS-IMPACT ANALYSIS AS A DECISION-AID 

Several advantages can be identified for the cross-impact 
technique when employed in assessing complex situations. 
First, its systematic approach assures that all problem 
elements identified as important are considered. The step 
by step procedure used can be easily traced to identify 
important elements in the problem situation, and to suggest 
areas requiring further study or close monitoring. 

Secondly, the technique can serve as a valuable heuristic 
device. Its systematic approach forces respondents to care- 
fully evaluate each estimate (opinion) in conjunction with 
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their other estimates , and to consider the estimates ' 
implications in toto. This requires that the experts fully 
and consistently formulate an underlying paridigm which can 
aid an individual or organization in problem definition. 

Thus, cross-impact analysis can be a useful modeling tool 
and problem specification approach. 

In addition, because the technique is time sensitive, 
its results can be used to monitor problem development, or 
at least the way in which the experts perceive that develop- 
ment. Through monitoring the development of cross-impact 
matrices it might also be possible to learn more about how 
individual, group, and organizational thought processes 
evolve in response to evolving situations, changing informa- 
tion streams, and differing perceptions of problem compo- 
nents. Thus, the technique's continued use might contribute 
to an understanding of how individuals process information 
(providing the data with which to resolve the question of 
how respondent's estimates should be interpreted), and how 
they perceive developing systems. 

Even without having completed the designed implementa- 
tion, because inconsistent ordered outcome probabilities 
were obtained, it is still possible to make some observa- 
tions regarding cross-impact analysis' utility as a decision- 
aid. Obviously, there are several problems which must be 
corrected before the technique can be applied in a "real- 
world" analysis. Many problems encountered in the 
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demonstration arose from a lack of experience in conducting 
subjective estimation sessions. Such "operator” problems 
included controlling the brainstorming session, and 
conducting the "event analysis." These problems were 
discussed, and solutions suggested in Chapter Four. None 
of the implementation problems appear to inhibit using the 
technique as much as the conceptual problem previously 
discussed. 

The theory that interactions among discrete events can 
be quantified and systematically assessed seems to be an 
appropriate method for approaching complex, ill-defined 
problems . The technique ' s demonstrated computer adapta- 
bility enhances this attractiveness. Using a computer 
makes it possible to identify a large number of events for 
analysis. In some cases, it might be possible to practically 
specify the entire problem environment. Although this would 
require a very large number of estimates, they could be 
obtained and manipulated with the computer. However, until 
the conceptual question discussed in the previous section 
is resolved, cross-impact analysis remains a developmental 
methodology . 
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APPENDIX A 



ESTIMATED RESPONSES 



1. Candidate event set, probability and significance 
ratings, impact scores (standard deviations) 



No. 


Event 

Statement 


Prob 


(S.D.) 


Sig. 


(S.D.) 


Imp. 


(S.D.)_ 


1 


Kurds in Soviet Union 
revolt. 


.115 


.08 


.48 


.31 


.62 


.27 


2 


Egypt closes Suez 
Canal to Soviets. 


.315 


.12 


.595 


.22 


.91 


.17 


3 


United States 
deploys Carrier Task 
Force to Indian 
Ocean. 


.755 


.19 


.62 


.19 


1.375 


.31 


4 


Soviet Union 
initiates military 
alert along 
Chinese border. 


.53 


.28 


.71 


.25 


1.24 


.47 


5 


Soviet Union block- 
ades Gulf of Oman. 


.275 


.19 


.725 


.25 


1.0 


.32 


6. 


Soviet Union mines 
St. of Hormuz. 


.185 


.14 


.77 


.25 


.955 


.26 


7 


United States appeals 
to all parties to 
cease hostilities. 


.805 


.13 


.!8 


.17 


1.04 


.17 


8 


United States offers 
to negotiate cease 
fire . 


.64 


.18 


.48 


.23 


1.12 


.27 


9 


United States evac- 
uates civilians 
from area. 


.835 


.10 


.305 


.14 


1.135 


.14 


10 


China ships arms to 
Baluchi rebels . 


.575 


.27 


.575 


.16 


1.15 


.39 


11 


India lends moral 
support to anti- 
Soviet forces . 


.685 


.14 


.405 


.22 


1.09 


.29 
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No . 


Event 

Statement 


Prob 


(S.D. ) 


Sig. 


(S.D.) 


Imp. 


(S.D.) 


12 


Chinese mobilize in 
Sinkiang. 


.505 


.25 


.715 


.20 


1.22 


.37 


13 


Iranian leftists 
sabotage oil fields. 


.485 


.16 


.715 


.16 


1.20 


.27 


14 


United States states 
it has no intention 
of becoming involved 
in the situation. 


.275 


.10 


.675 


.31 


.945 


.28 


15 


Leftists seize 
control in Iran. 


.37 


.14 


.805 


.14 


1.175 


.13 


16 


United Nations 
meets to discuss 
the problem. 


.885 


.08 


.195 


.14 


1.08 


.17 


17 


Iranian and Soviet 
units skirmish in 
area around 
Afghanistan . 


.485 


.25 


.625 


.20 


1.12 


.37 


18 


Iranian government 
clamps down on 
leftists . 


.71 


.18 


.375 


.13 


1.09 


.13 


19 


Civil War breaks 
out in Iran. 


.535 


.24 


.595 


.24 


1.22 


.18 


20 


United States 
convoys oil ship- 
ments with Naval 
units . 


.385 


.25 


.570 


.22 


.955 


•42 


21 


Ayatolla Khomeini 
calls for Jihad. 


.48 


.25 


.545 


.27 


1.03 


.27 


22 


Soviet backed Afghani 
forces subjugate 
Baluchistan. 


.595 


.15 


.885 


.14 


1.48 


.19 


23 


Soviet Union air- 
lifts supplies to 
Afghanistan . 


.72 


.20 


.65 


.17 


1.37 


.34 


24 


United States sends 
aid to Pakistan. 


.465 


.16 


.60 


.12 


1.07 


.24 


25 


United States sends 
aid to Iran. 


.425 


.18 


.58 


.21 


1.01 


.37 
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No. 


Event 

Statement 


Prob 


(S.D.) 


Sig. 


(S.D.) 


Imp. 


(S.D.) 


26 


United States 
deploys troops to 
Saudi Arabia. 


.255 


.13 


.625 


.17 


.88 


.22 


27 


United States 
deploys an F-15 
squadron to 
Saudi Arabia. 


.475 


.23 


.425 


.2 


.90 


.40 


28 


Soviet Union builds 
up its Indian Ocean 
squadron . 


.795 


.19 


.625 


.13 


1.42 


.28 


29 


Soviet Union 
harrasses oil tankers. 


.25 


.20 


.685 


.25 


.935 


.30 



2. 


Ranking by Impact (top ten) 






NO. 


All Events 
Short Name 


No. 


Response Events 
Short Name 


22* 


S.U. /Afghan forces gain control 


3* 


U.S. deploys CTF 


28* 


S.U. builds up I.O. sqdm. 


9* 


U.S. evacuates civilians 


3 


U.S. deploys CTF 


8* 


U.S. offers negotiation 


23* 


S.U. supplies Afghanistan 


7 


U.S. appeals for cease fire 


4 


S.U. mobilizes Chinese border 


24 


Aid Pakistan 


19 


Civil War in Iran 


20 


COnvoy oil 


12 


Chinese mobilize in Sinkiang 


25 


Aid Iran 


13 


Leftists sabotage oil in Iran 


14 


Non- involvement 


15 


Leftist coup in Iran 


27 


F-15's to Saudi Arabia 


10 


China arms Baluchis 


26 


Troops to Saudi Arabia 



(* indicates events chosen for analysis) 



3. Estimated interactions by Event Pair 



Event Pair 
( i / j) 

P(i) P(j) 


C 1 

(i| j) 


C 2 

(j|i) 


C 3 

(i| j) 


C 4 

(j|i) 


C 5 

( i f | i&j) 


(22,28) 
.6 .8 


. 66 


. 83 


. 56 


.80 


.23 


(22,23) 
.6 .72 


. 72 


o 

00 

• 


. 54 


.70 


.24 



Event Pair 
(if j) 

P (i) |P ( j) 


c i 

U|j> 


C 2 

( j 1 i) 


C 3 

(ill) 


C 4 

( j 1 1) 


C 5 

(i f 1 i& j ) 


(22,3) 
. 6 | . 76 


.57 


.85 


. 67 


.76 


.51 


(22,9) 
. 6 | . 84 


.68 


.92 


.59 


. 83 


.51 


(22,8) 
. 8 | .72 


.58 


. 61 


. 65 


. 68 


. 32 


(28,23) 

✓ 

.8 | .76 


.82 


. 77 


.79 


.72 


.56 


(28,3) 
.8 | . 84 


. 86 


. 87 


.80 


.75 


.61 


(28,9) 
. 8 | . 64 


.81 


.86 


.80 


. 84 


.77 


(28,8) 
.72 | .76 


.8 


.66 


.81 


.66 


.74 


(23,3) 

. 72 | . 84 


.75 


.81 


.73 


.75 


.56 


(23,9) 
.72 | .64 


.73 


. 86 


.72 


.83 


.78 


(23,8) 

. 76 | .84 


.73 


.70 


. 73 


.64 


.69 


(23,9) 

. 76 | .84 


.79 


. 86 


.78 


. 83 


.80 


(3,8) 

. 76 | .84 


.77 


. 65 


.77 


. 68 


.52 


(9,8) 
.84 | .64 


.82 


. 6 6 


. 85 


. 67 


.35 
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APPENDIX B 



CALCULATED RESPONSES 



. Resultant ordered outcome probabilities by Event Pair 



Event Pair 
(i/ j) 


P 1 

(i, j ) 


P 2_ 
(i/3 ) 


(i/ j) 


P 4_ 

(i/ j) 


P 5 

( j / i) 


P 6_ 

( j ,i) 


P 7 

( j ,i) 


!s_ 

( j / i) 


(22,28) 


2.6 


.53 


-9.7 


-2.4 


8.7 


4.5 


39.3 


30.9 


(22,23) 


5.1 


.53 


-15.3 


-7.4 


16.3 


6.3 


-66.6 


-56.8 


(22,3) 


1.6 


.28 


-2.5 


-.78 


1.5 


1.2 


-2.7 


-1.3 


(22,9) 


-1.5 


-.13 


-.6 


-.12 


-1.5 


-.69 


3.6 


2.5 


(22,8) 


4.5 


2.9 


-18.3 


-8.6 


6.1 


4.5 


-22.4 


-12.1 


(28,23) 


1.3 


.38 


-1.3 


-.51 


1.0 


.22 


-1.3 


-.35 


(28,3) 


1.2 


.12 


-1.2 


-.39 


.75 


.12 


-.55 


-.14 


(28,9) 


1.5 


.24 


-.55 


-.10 


.44 


.10 


.16 


.04 


(28,8) 


-.16 


-.1 


-.70 


-.36 


-.06 


-.02 


.91 


.21 


(23,3) 


1.4 


.32 


-1.4 


-.47 


1.1 


.37 


-1.5 


-.55 


(23,9) 


1.2 


.19 


-.57 


-.12 


.34 


.13 


.23 


.09 


(23,8) 


2.3 


.97 


-1.2 


-.67 


1.0 


.38 


-1.2 


-.44 


(3,9) 


1.1 


.18 


-.48 


-.10 


.28 


.07 


.37 


.08 


(3,8) 


1.9 


1.0 


1.8 


-.85 


1.7 


.50 


1 -3.4 


-1.0 


(9,8) 


1.2 


.63 


-2.7 


-1.3 


2.3 


.50 


-6.8 


.-1-2 
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2. Probability rankings by Event Pair 

(22,28) Or: Sov/Afg. forces gain control of Baluchistan, 

Soviet Union builds up 1.0. squadron 



Outcome Ranking Meaning 



1. 


P 7 


( j ,i) 


2. 


P 8 


( j , I) 


3. 


P 5 


( j /i) 


4. 


P 6 


( j /I) 


5. 


P 1 


(if j) 


6. 


P 2 


(i/ j) 


7. 


P 4 


(1/ j ) 


8 . 


P 3 


(If j) 


Evaluation: 


Inconsistent 



Pairing by Opp. Cause 



□ 



(22,23) Or: Sov/Afg. forces gain control of Baluchistan, 

Soviets airlift supplies to Afghanistan 



Outcome Ranking Meaning 



1 . 


P 5 


( j f i) 


2. 


P 6 


( j ,1) 


3. 


P 1 


(i/ j) 


4. 


P 2 


(i/ J) 


5. 


P 4 


(If j) 


6 . 


P 3 


(I, j) 


7. 


•d 

CD 


( j f I) 


8. 


? 7 


( j » i) 


Evaluation : 


Consistent 



Pairing by Opp. Cause 
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(22,3) Or: Sov/Afg. forces gain control of Baluchistan, 

U.S. deploys CTF to 1.0. 



Outcome Ranking Meaning 



Pairing by Opp. Cause 



1. 


P 1 


(i, j) 


2. 


P 5 


( j ,i) 


3. 


P 6 


( j ,1) 


4. 


P 2 


(if J) 


5 . 


P 4 


(if j ) 


6 . 


P 8 . 


(3 f I) 


7. 


P 3 


(If j ) 


8. 


P 7 


(3fi) 


Evaluation : 


Apparently I 




Dif , 



= .2 



from rank 7 to rank 8 is only .2; therefore, 
possibly consistent 



(22,9) Or: Sov/Afg. forces gain control of Baluchistan, 

U.S. evacuates civilians from area 



Outcome Ranking Meaning 



1. 


P 7 


( j f i) 


2. 


P 8 


( j f i) 


3. 


P 4 


(If 3) 


4. 


P 2 


(if 3) 


5. 


P 3 


(If j) 


6. 


P 6 


( j f I) 


7. 


P 5 


( j f i) 


8. 


P 1 


(if j) 


Evaluation : 


Inconsistent 



Pairing by Opp. Cause 



□ 
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(22,8) Or: Sov/Afg. forces gain control of Baluchistan, 

U.S. offers to negotiate a cease-fire 



Outcome Ranking Meaning 



1. 


P 5 


( j ,i) 


2. 


P 1 


(if j ) 


3. 


P 6 


CJ. 

1 HI 


4. 


P 2 


(i/ j) 


5 . 


P 4 


(If 5) 


6 . 


P 8 


( j f I) 


7. 


P 3 


(If j ) 


8. 


P 7 


(If i) 


Evaluation : 


Consistent 



Pairing by Opp. Cause 




(28,23) Or: S.U. builds up 1.0. squadron, 

Soviets airlift supplies to Afghanistan 



Outcome Ranking Meaning 



Pairing by Opp. Cause 



1. 


P 1 


(if j ) 


2. 


P 5 


( j f I) 


3. 


P 2 


(ifl) 


4. 


P 6 


(If i) 


5. 


P 8 


(Ifl) 


6 . 


P 4 


(ifl) 


7. 






8. 


TIE P 3 & P ? 


(I, j) 


Evaluation: Consistent k 



TIE 
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(28,3) Or: S.U. builds up I.O. squadron 

U.S. deploys CTF to I.O. 



Outcome Ranking Meaning 



Pairing by Opp. Cause 



1 . 

2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 



TIE P 0 & P- 
2 6 



(if j ) 

( j /i) 

(if j) & (jfl) 

(I f I) 

(If j) 

( j f i) 

(If j ) 




TIE 



Evaluation: Consistent due to tie as in previous case. 



(28,9) Or: S.U. builds up I.O. squadron, 

U.S. evaculates civilians from area 



Outcome Ranking Meaning 



Pairing by Opp. Cause 



1. 


P 1 


(if j ) 


2. 


P 5 


( j f i) 


3. 


P 2 


(if j) 


4. 


P 7 


(I f i) 


5. 


P 6 


(jfl) 


6 . 


P 8 


(5 f I) 


7. 


P 4 


(If j) 


8. 


P 3 


(If j ) 


Evaluation : 


Inconsistent 
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(28,8) Or: S.U. builds up I.O. squadron 

U.S. offers to negotiate a cease-fire 



Outcome Ranking Meaning 



1 . 


P 7 


(If i) 


2. 


P 8 


(If I) 


3. 


P 6 


( j f I) 


4. 


P 5 


( j f i) 


5 . 


P 2 


(if j) 


6. 


P 1 


(if j) 


7. 


P 4 


(i f j ) 


8 . 


P 3 


(if j) 


Evaluation : 


Inconsistent 



Pairing by Opp. Cause 

□ 



(23,3) Or: S.U. airlifts supplies to Afghanistan, 

U.S. deploys CTF to 1.0. 

Outcome Ranking Meaning Pairing by Opp. Cause 



1. 


P 1 


(i f j) 


2. 


P 5 


( j f i) 


3. 


P 6 


( j f I) 


4. 


P 2 


(if I) 


5. 


P 4 


(if j) 


6 . 


P 8 


( j f I) 


7. 


P 3 


(I, j ) 


8. 


P 7 


(If i) 


Evaluation: 


Apparently I 



Consistent . 




Dif . = .07 
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(23,9) Or: S.U. airlifts supplies to Afghanistan, 

U.S. evacuates civilians from area 



Outcome Ranking Meaning 



1. 


P 1 


(i , j ) 


2. 


P 5 


( j , i) 


3. 


P 7 


( j ,i) 


4. 


P 2 


(i /• j ) 


5. 


P 6 


( j >1) 


6 . 


P 8 


( 3,1) 


7. 


P 4 


(i, j ) 


8 . 


P 3 


(I, j) 


Evaluation : 


Inconsistent 



Pairing by Opp. Cause 

II 

EH 



(23,8) Or: S.U. airlifts supplies to Afghanistan, 

U.S. offers to negotiate a cease-fire 



Outcome Ranking Meaning 



Pairing by Opp. Cause 



1 . 


P 1 


(if j) 


2. 


P 5 


( j r i) 


3. 


P 2 


(if 3 ) 


4. 


P 6 


( j f I) 


5. 


P 8 


( j ,1) 


6. 


P 4 


(If D) 


7. 


P 3 


(I, j) 


8. 


? 7 


( j ,i) 


Evaluation : 


Apparently I 



Dif . I = 



difference between rank 7 and rank 8 is only 
.01, therefore, Consistent. 



.01 
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(3,9) Or: U.S. deploys CTF to 1.0. 

U.S. evacuates civilians from area 



Outcome Ranking Meaning 



1. 


P 1 


(i/ j) 


2. 


P 7 


(Li) 


3. 


P 5 


( j / i) 


4. 


P 2 


(i /I ) 


5. 


P 8 


( j /I) 


6 . 


P 6 


l*H 1 

•m | 


7. 


P 4 


(i, j ) 


8. 


P 3 


(i/ j) 


Evaluation : 


Inconsistent 



Pairing by Opp. Cause 

□ 

□ I 



(3,8) Or: U.S. deploys CTF to I.O., 

U.S. offers to negotiate a cease-fire 



Outcome Ranking Meaning 



1. 


P 1 


V. 

*H 1 


2. 


P 3 


(i/ j ) 


3. 


P 5 


( j /i) 


4. 


P 2 


(i, j) 


5. 


P 6 


( j ,1) 


6. 


P 4 


(I. j) 


7. 


P 8 


( j / i) 


8. 


P 7 


( j / i) 


Evaluation: 


Inconsistent 



Pairing by Opp. Cause 

3 
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9,8 Or: U.S. evacuates civilians from area, 

U.S. offers to negotiate a cease-fire 



Outcome Ranking Meaning 



1 . 


P 5 


( j , i) 


2. 


P 1 


(i/ j ) 


3. 


P 2 


(i, j) 


4. 


P 6 


( j /I) 


5 . 


P 8 


(5,1) 


6 . 


P 4 


(1,3) 


7. 


P 3 


(i, j ) 


8. 


P 7 


(j ,i) 


Evaluation : 


Consistent 



Pairing by Opp. Cause 



3. Summary of Results 



Case Pair Consistent Order 



1 


(22,28) 


NO 


7, 


8, 


5, 


6 , 


1, 


2, 


4, 


3 


2 


(22,23) 


YES 


5, 


6, 


1, 


2, 


4, 


3, 


8, 


7 


3 


(22,3) 


NO 


1, 


5, 


6, 


2, 


4, 


8, 


3, 


7 


4 


(22,9) 


MO 


7, 


8, 


4, 


2, 


3, 


6, 


5, 


1 


5 


(22,8) 


YES 


5, 


1, 


6, 


2, 


4, 


8, 


3, 


7 


6 


(28,23) 


YES 


1, 


5, 


2, 


6, 


8, 


4, 


7, 


3 


7 


(28,3) 


YES 


1, 


5, 


2, 


6, 


8, 


4, 


7, 


3 


8 


(28,9) 


NO 


1, 


5, 


2, 


7, 


6 , 


8, 


4, 


3 


9 


(28,8) 


NO 


7, 


8, 


6, 


5, 


2, 


1, 


4, 


3 


10 


(23,3) 


YES 


1, 


5, 


6 , 


2, 


4, 


8, 


7, 


3 


11 


(23,9) 


NO 


1, 


5, 


7, 


2, 


6, 


8, 


4, 


3 


12 


(23,8) 


YES 


1, 


5, 


2, 


6, 


8, 


4, 


7, 


3 


13 


(3,9) 


NO 


1, 


7, 


5, 


2, 


8, 


6 , 


4, 


3 


14 


(3,8) 


NO 


1, 


3, 


5, 


2, 


6, 


4, 


8, 


7 


15 


(9,8) 


YES 


5, 


1, 


2, 


6, 


8, 


4, 


3, 


7 



All inconsistent outcome orders are unique. 

The seven consistent outcome patterns occur in 5 sets. 
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